Subbu

KV Cache re-use is the most important thing for agentic rollouts. We've integrated Mooncake Store into prime-rl with vLLM, you can now use it as a drop-in replacement for native CPU/Disk offloading, giving you cross-node prefix cache reuse to make your agents go brrr🚀

Local inference not looking so crazy anymore now that companies are revolting over token spend

IMPORTANT: it is important to understand that the CoreWeave & Microsoft photos are still Engineering/Quality Samples, and there is still some time before the software stack bring-up finishes & first production tokens are generated. The VR200 & MI455 rack metric to watch out for is time to first at-scale production token TTF-(ASP)-T. You can clearly see in the CW rack photos that none of the scale-out 800G OSFP cages are even populated.

My 2nd biggest position btw I don’t have a Substack, but if I did people would be very happy with me

I think Anthropic likely trained the Mythos base model from roughly October to December using on the order of 6.7e26–1.0e27 flops Since then, the RL-to-base-model-training flop ratio is plausibly somewhere around 0.5 and 3, depending on how much of the expanded Trainium 2 fleet was actually allocated to Mythos RL. The reason this range is plausible is that public AWS/Anthropic statements imply Anthropic-accessible Trainium2 capacity grew from roughly 500k chips around Rainier’s launch to over 1M Trainium2 chips for Claude training and serving.

If a baker makes a cake with their own recipe using ingredients from a mutual friend, then gives it to you, would you pass it off as your own? Mediatek makes a CPU with their own recipe with arm ingredients and gives it to Nvidia. Broadcom makes a TPU with their own recipe but with Google ingredients and gives it back to Google.

21 petabytes of memory bandwidth.