Tiberiu Mușat

🚨 New paper at ICLR 2025 on LLM Interepretability Why do LLMs need so many layers? 👀 👇 Key findings from my latest paper "Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers" (1/6)

@BushnaqLucius arxiv.org/abs/2506.12027

@Tiberiu_Musat_ For the first two, don't you run into trouble because your activations and parameters are finite float precision, meaning the positional encoding in the attention mechanism doesn't work anymore past a certain size? Or are we macgyvering some kind of different access mechanism?

Why does deep learning generalize? What does weight decay really do? Can algorithmic information theory address these questions? In my latest preprint, I give a proof that the minimum neural weight norm matches the minimum program length (aka Kolmogorov Complexity), up to a logarithmic factor. In other words, the neural network with the smallest possible weight norm (that fits the data) must encode the shortest program (that fits the data). The result only holds for fixed-precision neural nets: infinite precision nets can store infinite information with finite (small) weights. arxiv.org/abs/2605.10878

@BushnaqLucius The proof works for looped neural architectures that can access an unbounded tape. Examples include chain-of-though transformers, looped transformers with large context, neural computers, etc.

@Tiberiu_Musat_ Actually wait I'm confused, the paper seems to say the result holds for vanilla K-complexity, not K-complexity under some kind of memory bound. How does that work memory-wise when the mlp has a fixed finite width?

@BushnaqLucius I agree, implicit biases usually induce a similar prior without explicit regularization.

@Tiberiu_Musat_ Nice. Notably, the proof is actually about the number of non-zero parameters in the network. So this solution would also be favoured in the training prior on account of its size in the loss landscape, not just because of explicit weight regularisation.

@Andres_Nava_12 @MatthieuWyart Very interesting! Seems related to arxiv.org/abs/2601.19208 @SharonYixuanLi

Very happy to share my first first-author preprint with @MatthieuWyart. Meaning is hierarchical: dog → mammal → animal. This hierarchy appears as geometry in LLM embeddings. But where does that geometry come from? We show that word co-occurrence statistics are sufficient to induce it. arxiv.org/abs/2605.23821

Mechanistic interpretability aspires to be the biology of deep learning. @KuninDaniel and @learning_mech say that an emerging theory of deep learning they and their team call 🛠️ learning mechanics 🛠️ will be the physics.

@uzaymacar Amazing work! 👏🏻

🧵New Anthropic Fellows research: We studied mechanisms of "introspective awareness" in LLMs. LLMs can sometimes detect steering vectors injected into their residual stream. But is this worthy of being called introspection, or attributable to some uninteresting confound?👇

LLM pretraining may follow a hidden curriculum. Across 9 open models, component skills tend to emerge before composite skills in a predictable order. The order is legible enough to predict held-out task trajectories. What this tells you is that a loss curve dashboard is not enough when training models. We need training milestone evals too, so you know where in the curriculum the model is.

(11/11) Try it in your browser! Works on mobile too. Link in my bio.

10/11

A few more photos from my Mandelbrot Set fractal explorer👇(1/11)

@GaspardProd Thank you! :)

@Tiberiu_Musat_ Awesome