ThierryLabro retweetledi
🚨 BREAKING: China just fixed a 10-year-old flaw hidden inside every major language model.
Every AI you use today (ChatGPT, Claude, Gemini) is built on a massive flaw. It’s called the residual connection.
Here’s the problem: every layer inside an AI blindly stacks its output on top of the last one. There is no filtering, no judgment, just blind accumulation.
Imagine a meeting where every person shouts their ideas at full volume, forever. The early ideas (the fundamental patterns) get drowned out by the newer, louder layers piled on top.
The technical term is “prenorm dilution,” but in practice, it means your AI forgets its own most important work as it gets deeper.
We’ve been building models like this since 2017.
Now, Moonshot AI (Kimi) just dropped a paper that completely replaces this broken system.
They call it “attention residuals.”
Instead of blindly accumulating everything, each layer now “votes” on which previous layers actually matter using softmax attention over depth. The network learns to remember what’s important and ignore what isn’t.
The results are absolutely insane:
- It matches the performance of models that used 25% more compute to train
- Tested on a 48-billion-parameter model with massive gains in math, code, and reasoning
- Inference slowdown is less than 2%
- It’s a direct drop-in replacement for existing systems
The Transformer replaced recurrence with attention across words in 2017. This paper is doing the exact same thing across layers of depth.
This is the same class of idea. The entire architecture of AI is about to change.
English
