super tender retweetet
The market is worried about memory because of $GOOGL's TurboQuant compression of the KV Cache shows you how little investors know about specific technical changes and what they actually mean. Some of my thoughts on this topic:
1. The TurboQuant concept is not new; it was already published one year ago as an arXiv (April 2025).
2. The paper has benchmarked the 8x performance increase in computing attention when there is an FP32 setup and a 6x memory reduction assessment on the FP16 baseline. Frontier AI labs do not run FP32/FP16 KV cache for inference in production today. Most of the inference on leading AI labs is run on FP8, some even FP4, so the claimed savings on HBM are much lower if you compare them to actual current productions. Also, every frontier AI lab is already compressing their KV cache as much as they can before this paper was published (this is nothing new in terms of the direction of the market). So the savings compared to what is being used mostly today are smaller than the 8x and 6x numbers.
2. When we get some savings on memory (they are happening all the time, through different iterations), we get better models that are able to serve a bigger context window. The difference in quality between a model with a 1M context window and a smaller context window model is enormous. This time will be no different; usage will grow even more as a result of this, as models get better and are able to have bigger context windows, and HBM demand will continue to grow (even for inference, where these optimizations are taking place).
4. Funny enough, with DeepSeek's Multi-head Latent Attention introduction (back in Jan 2025), the KV cache was compressed by a lot more than this and still resulted in drastically more memory demand.
English
