nahcrof

where else can you get 500 daily requests on a model like glm-5 or kimi-k2.5 for $5 (without extra rate limits)?

@vibestressing About 4-6 hours somewhere between 00:00 and 10:00 CST

@nahcrof everything you do is very impressive. so, when do you sleep?

I have been gifted the silly blue checkmark

@Star_Knight12 CrofAI in my biased opinion

what's the best thing happened because of AI

@viktorg475 @bianco_____ I don’t adjust the model, it’s the inference engine and quantization, I can’t control much but I can always do my best

@nahcrof @bianco_____ A bit of curiosity on how this all works, why is this your responsibility and not Zhipu's? Aren't you just using their weights? How can you make "adjustments" to an already trained model?

So far, any feedback on the improved glm-5.1 deployment?

@bianco_____ Alright, I’ll admit I haven’t messed with spawning sub agents so I’ll look into it, thank you for the feedback

@nahcrof having a hard time making it spawn subagents. i have to ask specifically for the task tool, it`s not smart enough to understand: "spawn 2 subagents ... " , while in fireworks and opencode providers, GLM 5.1 gets it really well.

@weirdLSD That’s the hope, we’ve been doing this for nearly a year now

@nahcrof will this still be true in a few years from now 😁

cheapest LLM inference world wide

@gidian83427 I mean I think that looks pretty good

@viktorg475 The tokens are counted by the tokenizer for the model so that’s why there’s variance, as for logging that is some time thing I intend on improving that and glm-5.1 should be better now, infra was struggling to keep up until today when I pushed a patch

So I bought some tokens and tried to use your service (glm-5.1) and nothing ever came back after a minute. You need better logging (like Openrouter has). And maybe average relative token expense to your cheapest model (ie: "what model are you?" in minimax is N tokens but N*4 in GLM)

Just realized that chutes tried to lower their kimi-k2.5 prices again so I’ll be lowering ours with our next update $0.35/m input $0.07/m cache $1.70/m output

@JaidCodes Yes, they actually reached out to me at one point and long story short they decided they didn’t want to (because of a bug that wasn’t our fault)

@nahcrof Did you ever try to apply to OpenRouter as an inference provider?

@guillermode20 I could probably do a $20 plan

@nahcrof Awesome! Any plans for a $20 plan? The jump from 10 to 50 is pretty significant. Either that or an option to buy packs like how synthetic does it would be cool.

glm-5-lightning is deprecated (reroutes to glm-5.1) the lightning infra that was going to glm-5-lightning is currently being moved to glm-5.1

(and sometimes fastest depending on the model)

@0oAstro You're not stupid! And thank you! I do my best

@nahcrof rlly sorry, i just re-did the setup and `crof.ai` worked instead of `crof.ai/v1`. should have tried earlier. i feel stupid now. + earlier it might be down as u were changing inference engine. great service and great work man :D

Update coming soon glm-5-lightning will be deprecated in favor of increasing overall inference tps on glm-5.1 (if you have tools that go to glm-5-lightning, once the update is released they will be redirected automatically so your tools don't break)

@oneabdulshakoor yes, the infra is the same no matter what

@nahcrof Does the api and subscription plan both have same infra so technically speed would be same, as chutes has been slow and opencode go hits limit very fast

where else can you get 500 daily requests on a model like glm-5 or kimi-k2.5 for $5 (without extra rate limits)?

@oneabdulshakoor I have been looking into adding that model, I just don't wanna use more capacity than I can comfortably handle

@nahcrof Also stepfun , probably will push me to switch once stepfun 3.5 is available its in top5 and benchmark are crazy for such a small model , just check openrouter usage , i have been following for a while and thinking of switching might try api first to test the speed

@0oAstro we do have a /v1/models endpoint but if you're having issues I can do my best to help

@nahcrof ahh yes, bifrost didn't seem to find /v1/models. do u already have it? in which case it might just be issue in my bifrost stack or something

@adhtri001 I have thought about speculative decoding but I haven't put much effort into it, I might do it though (especially for kimi-k2.5)

@nahcrof Hey, was wondering if you wanted to consider eagle3 speculative decoding? Generally, offers 3x throughput, without any quality loss. Could do 5x in rare cases. As of now, I think k2.5 have a good eagle3 model in HF.

Should we replace glm-5-lightning with glm-5.1 lightning or get rid of glm-5-lightning and aim to keep glm-5.1 tps above 100?