Sabitlenmiş Tweet
Okay guys, FINALLY follow-up on the HBM stress test. because I cracked my head on this topic way harder then I ever tought to do. Put so much time in this engine and learned also so much. So must read here!
Besides all the supply constraints that i mapped in other parts which are already as tight as it can be I looked at demand now and how it develops.
So four levers drive the supply gap. I pushed each to its worst realistic setting, one at a time, and the shortage held every time… except one lever could flip it alone. Efficiency. How fast the industry learns to serve more AI per gigabyte of memory.
So we measured it. Properly.
First thing to understand… our base case already assumes brutal efficiency. Tokens are growing 4 to 7x a year (Google confirmed 7x at I/O), yet our memory demand line only grows ~40%. That wedge IS efficiency… roughly 30% per year, already priced in. MLA cutting KV-cache roughly 3 to 5x. FP8 to FP4 quantization. Sparse attention. All of it, already in this number.
To flip 2027–2028 into surplus, efficiency would need to run another ~25% per year ON TOP of that 30%.
The measured incremental rate? Around 5% a year. And decelerating… the big wins were one-time steps, mostly shipped in 2024–2025, and we are approaching hard floors. 4-bit precision. ~70KB per token on KV-cache.
Then the other escape hatch… “CXL pooling and flash tiers will fix it.” I ran that one too. Most memory tier below HBM serves the COLD side. CXL pools sit at hundreds of nanoseconds… good for cached prefixes and reuse. High-bandwidth flash is read-mostly, write endurance caps it at shared context. System RAM costs 10 to 50 milliseconds per fetch. But the hot decode set… the memory a GPU touches every single token… is pinned to HBM by latency. NVIDIA even built a separate GDDR7 chip for the part that can leave (Rubin CPX, prefill). The boundary is drawn in silicon. These tiers are additive, not substitutive. They grow what is servable… they do not shrink what needs HBM.
For the math people: balancing 2028 would need roughly three quarters of KV traffic off HBM. The physical ceiling is nowhere near that.
And here is the stat that settles it. No GPU generation has EVER shipped with less memory per system. 80GB… 141… 180… 288 and rising. Every byte the software saves gets spent immediately on longer context and bigger batches. Agent tasks now burn millions of tokens each. That is Jevons paradox in silicon.
Now the honest part, because there always is one. Nobody publishes memory-per-token over time, so the 5% is our inference… wide error bars. And I wrote down exactly what would change our mind: production KV compression clearing 5x+, a generation shipping lower memory per rack, or the flash tier proving it can hold the hot set. Low probability for all.
in my opinion the two escape routes everyone cites… efficiency, and moving memory off HBM… I measured. Neither runs fast enough.
$MU $005930.KS $000660.KS
English
