Taelin@VictorTaelin
> Read this and your next model will be 10x smarter. <
Nobody knows what intelligence truly is. We just know models are converging to being smarter, as they train. Yet, we DO know some of the fundamental features of intelligence. And when one of these features is neglected or not trained for, then there is no way for a model to obtain it. Neglecting an aspect of intelligence hinders a model's general capabilities, in a way no amount of flops can compensate for.
I'm making this whole post to convince you there is ONE fundamental aspect of intelligence that YOU are neglecting, underestimating, and under-training for. Anyone using models 24/7 can see this weakness. It is blinding, glaring, as clear as skylight.
That feature is: ✨ erasure ✨
Removal. Compression. Garbage collection.
Models are not sufficiently trained for that.
They are trained to ADD information.
Not to REMOVE it.
You ask a question. They give you an answer.
They work in a project. They write files.
You post a bug. They craft a solution.
They're only indirectly, if at all, rewarded for removing information, or compressing information. This is a huge mistake, because erasure is a cornerstone of intelligence.
The human brain has several mechanisms entirely dedicated to removing information. Short term memory, long term memory, sleep, all mechanisms to throw garbage away. Furthermore, grokking is nothing but a compression event. An aha-moment happens when your brain is capable of expressing new information in terms of information you already posses stored. This is what allows that info to be stored. That is how you learn.
Erasure isn't a small feature, erasure is *THE* underlying driver of intelligence. It is what allows us to keep absorbing tons of information and still managing to turn it into useful capabilities. Intelligence is not about producing good knowledge, it is about removing bad knowledge. So, erasure is half of it.
So, my advice to you: take erasure seriously. Train on it. The architecture is fine. It can lead to AGI. But you won't be a complete athlete if you skip leg day. Reward your model on the other half of intelligence. Teach it how to erase and compress information competently. Make this a big program in your company. Have entire teams dedicated to this.
"I'm already kinda doing that!"
No you are not. And if you think you are, take this as a signal you should do 100x more of it. I want to be very clear here: erasure is HALF of intelligence. So, if not half of your FLOPS are flowing into erasure, you're wasting your GPUs, and no optimizer can compensate for that.
"But how do I teach a model to erase?"
Literally, just ask it to compress a text, then reconstruct it, and ask questions to assert how lossless the conversion was. That simply. You can do that in any dataset.
For coding, a more effective way is to take a big codebase and ask it to make it shorter, while still preserving the same behavior. IMPORTANT: avoid code-golfing / minification / uglification. Removing comments or making variable names shorter IS reward hacking. Counter that by counting the NUMBER OF BRANCHES. A branch is: an "if", a "match", a "case". That's THE complexity of your program.
Count it, and ask the model to reduce it. There's no way to do so other than building better abstractions. And a better abstraction is nothing more than a blob of information that lets you throw other information away, because it expands into the information that was just discarded. Train on that, and your model will be incentivized to build better abstractions. Do you know what we call humans capable of building better abstractions? Geniuses.
So, please: appreciate the full nature of intelligence and give your models the rewards they need to train on all of it. Let erasure be a major part of your training programs. Do not skip leg day.
Thanks for coming to my TED talk...
