Jasper Gilley

# Why RL on high-dimensional data recreates Zipfian grokking dynamics Suppose a model's learned representations form a manifold M of intrinsic dimensionality k_model, and the task has intrinsic dimensionality k_task. If k_model ~= k_task (i.e., tasks like math and coding), then policy gradients will tend to be on-manifold. But if k_model << k_task (i.e., open-ended reasoning, creative writing), RL recreates Zipfian grokking dynamics. We expect advantages to be heavy-tailed because they are created by sequential composition where good choices (actions, next token predictions, etc.) compound on each other. The highest-advantage rollouts are therefore the ones that compounded the most atypical choices, pushing their activations furthest from M. And because arbitrary displacements in high-dimensional space are increasingly likely to be orthogonal to any given manifold, these large displacements are overwhelmingly off-manifold. Off-manifold updates are tantamount to memorization because they reject the directions that the model has already chosen to compress the data. So, the highest-advantage rollouts lead to the most memorization — and they carry the most gradient weight. Therefore, RL in high-dimensional space is likely to create the "soft ceiling on abstraction quality" I refer to in the article :)

@YouJiacheng @cloneofsimo It's also got to be a matter of the intrinsic dimensionality of the data, language is uber compressed

@cloneofsimo I wonder if we train 100 epoch with 100TPP data, what will happen.

really? are we data-rich?

@RoyalCities Most exciting work going on in music AI in a long time! You rock, keep it up

As just some random dude, seeing my model hit top trending on Hugging Face is kinda insane to me. It’s been about a week, so here’s what worked, what didn’t, and what’s coming next 👇 (short 📜- spoilers I’m not done)

@_ueaj Right! So it seems like perhaps the simplest explanation for at least part of the phenomenon of scaling laws is the heavy-tailed nature of the things we're training on?

well my point is that *everything* has zipf distributions (or more precisely, pareto distributions). Hierarchical network connections are also distributed like this, and occur everywhere (neurons in the brain, internet hyperlinks, computer networks, food chains, etc.) The density distribution of matter in our universe is also a power law. So is the distribution of atomic masses of compounds. Literally *everything* at sufficient scale in the universe is organized like this

smth really cool I learned at pangram is that AI detection has scaling laws too both for sample context length and parameter count. Haven't quantified it exactly but it's def there

@_ueaj I mean yeah but images, user behavior, audio etc. also happen to have Zipf-like distributions. You could do similar papers to the above in those domains too

Actually scaling laws apply to images, video, audio, embeddings, recsys, agentic behavior, etc. it's not just language. IMO there are 2 even more fundamental laws of physics deeper than the literal QM/GM rules. 1. hierarchical organization 2. emergent complexity The deep neural network is the perfect inductive bias for emergent complexity, as the circuits deeper in the model can be composed into higher level ones, i.e. they affect the hypothesis space of higher layers, which is exactly how emergent complexity works. This is why DNNs work in virtually every field at scale: it's the only truly universal inductive bias

@_ueaj I imagine it owes to the natural distributions of natural language! No reason it'd make a difference whether you're modeling it or meta-modeling it arxiv.org/abs/2602.07488

I can first principles explain why scale works everywhere but it's cool to see it validated experimentally

Amazing post that deserves much more attention... Confirms the intuition that your data distribution acts as a generalisation pressure/regulariser through amazingly clean toy experiments... Really recommend reading it.

@hdarshane 🙏😄

@truththrulove @WilliamShatner You're a fake Trek fan if you think this, Discovery notwithstanding

@WilliamShatner Original Trek was written as a conversation. New Trek is written as a lecture.

During the first airing of my Star Trek series where a kiss was objectionable; many southern stations pulled the episode & condemned the show. Using today’s vernacular it would absolutely be called“woke DEI crap”because it went against “norms” of society for its time. Not a lot seems to have changed.🤷🏼😑

@_TobiasLee x being good enough depends on models having a privileged view of their own internals

continual learning for agents can be explained in one line: Y = Wx - change x: memory.md - change W: weight updates, e.g., LoRa change x is sufficiently good now

@hdarshane agreed on all accounts! It definitely feels like a poorly-motivated/poorly-executed test of generalization. but you would probably still concede that they did show something like "the model's programming reasoning circuitry is somewhat entangled with its Python circuitry", right?

@0xjasper not in good spirit to shackle generalisation that happens via those vectors, thus reducing total generalisation capabilities, and to then say something like "these models fail to generalise" in the Gary Marcus dialect or whatever

keeps getting worse so unserious

@Noahpinion IMO the proper term for this sort of thinking is more 'leftist' than 'progressive'

Yes. Progressive culture frames relationships as being all about power and extraction instead of love, affection, etc. It's horrible.

@rabrg @eshear You can compress lots of uninteresting things without actually learning anything though!

@eshear in other words, compression!

Learning is as much about effectively forgetting noise as it is about remembering signal.

@joelithic @tenobrus Peanut brain ahh take

@0xjasper @tenobrus Gavin has been terrible for California.

i can't fucking believe this is really our roster man. what the fuck did we do to deserve this

I suspect that the path to superintelligence lies through the sort of thing that could be called "gradient aesthetics" If you've already learned from all text ever written, the only way to get more signal is to think harder about the best data you have

@yearemias @tszzl @Noahpinion Try it yourself

@0xjasper @tszzl @Noahpinion source?

The reason AI still can't write well is that it writes what IT wants to say, not what YOU want to say. Writing is thinking. I expect this to be fixable, but not via typical "scale to AGI" approaches.

@0xHondo @tszzl @Noahpinion No

@0xjasper @tszzl @Noahpinion thats a feature, not a bug

@tszzl @Noahpinion The biggest models behave the same way though, for the most part

@Noahpinion “using gpt 5 mini” every. damn. time