Marc retweetledi
Google's latest paper on Compression is the future.
Here's why.
They compressed LLM memory 6x with zero accuracy loss.
When ChatGPT writes a reply, it remembers every word you've said. That memory is stored in a growing notebook (KV cache).
A 100,000-word conversation can eat 16 GB of GPU memory. That's half of what most high-end GPUs even have.
This is the #1 cost of running AI. Not the thinking. The remembering.
TurboQuant shrinks each number in that notebook from 32 bits to just 3.
That's like replacing a full paragraph with three words and losing nothing. No retraining. Works on any model instantly.
Compressing numbers usually destroys their meaning. Here's how they solved it:
1. Rotate the numbers randomly so they all land on a predictable curve (PolarQuant)
2. Use one extra bit to fix the tiny errors left behind (QJL)
Once numbers are predictable, you need far fewer bits to store them.
The results:
> 8x faster on Nvidia H100 GPUs
> 16 GB notebook shrinks to under 3 GB
> Search indexing drops from 500 seconds to 0.001
> Accuracy identical to the uncompressed model
There's a proven math limit on how good compression can get. TurboQuant is only 2.7x above that floor. We're near the ceiling.
Every company running LLMs spends most of its budget on memory. This cuts that cost by over 80%.
The race is no longer about bigger models. It's about cheaper inference.
Models that needed a $200K server cluster start fitting on a single $2K GPU. AI agents run 24/7 without burning budgets.
The companies that win won't just have the best models. They'll have the best compression.
Papers are open-access on arXiv, presented at ICLR.
English
