Daniel Kunin

For the last few years, a lot of my work has been driven by the feeling that deep learning is not magic — there are principles, mechanisms, and laws waiting to be understood. This paper is our attempt to say that clearly!

A longstanding dream of interp is to decompose activations into distinct, interpretable parts. But when should we expect that to work, and what even are such parts? New from Simplex: transformers factor their world into orthogonal subspaces, even when it costs accuracy.🧵👇

@rzebrahimi @giovannimarchet @AdeleMyersPhD @hopfbifurcator @ninamiolane @rzebrahimi this is a very interesting and related work -- lets chat!

@KuninDaniel @giovannimarchet @AdeleMyersPhD @hopfbifurcator @ninamiolane Nice! This seems very related to our work too: we empirically compare transformers and RNNs on group composition tasks: arxiv.org/abs/2602.18333

Excited to share that our paper “Sequential Group Composition: A Window into the Mechanics of Deep Learning” was accepted to ICML 2026 in Seoul! Co-led with @giovannimarchet and @AdeleMyersPhD @hopfbifurcator @ninamiolane Paper: arxiv.org/abs/2602.03655

For me, this paper is learning mechanics in action! Mech interp first identified that NNs use Fourier features in algebraic tasks - great work @bilalchughtai_ @justanotherlaw @NeelNanda5 Learn mech asks why training produced those features, in that order, with that architecture

Yes — by leveraging associativity. We explicitly construct efficient solutions: RNNs can compose elements sequentially in k steps, while deep MLPs can compose adjacent pairs in parallel in log k layers and we find preliminary evidence that GD can discover these solutions!

From "Mathematical theory of deep learning: Can we do it? Should we do it?" to "There Will Be a Scientific Theory of Deep Learning". It's respectively the title of a talk I gave four years ago, and the title of an arxiv paper from four days ago. I really like the "learning mechanics" perspective (think of it as a continuation of "statistical mechanics", "quantum mechanics", and so on). Several of my last academic papers can be viewed under that lens (e.g. Learning threshold neurons via the “edge of stability”; or LEGO). I'm not as optimistic as the authors of the recent arxiv paper that we will EVER be able to reach what the "physics mechanics" field have achieved, but it's certainly worth trying. Talk: youtu.be/3uRD_lg701k?si… Paper: arxiv.org/abs/2604.21691

@scifordl @ZKadkhodaie @FlorentinGuth @learning_mech @AntSclocchi Love the hats!

Organizers organizing

@tydsh Yes, agreed. “Learning mechanics” is a scientific program many researchers have been building toward for years; our goal was to clearly state the evidence for such a theory Your works are great examples of 2.1 (have read many), we should have cited them, adding them now for v2

History repeats itself 😀 The concept "learning mechanics" is not a new thing but actually has been explored for a very long time. It is human nature to think deeper than a blind belief of the scaling laws. I have been working on rigorously modeling training dynamics of deep nonlinear models for many years, with many non-trivial solvable examples in nonlinear dynamics that may be interesting for @learning_mech to take a look. This includes 1. Contrastive learning (e.g., arxiv.org/abs/2110.09348 arxiv.org/abs/2201.12680, arxiv.org/abs/2206.01342) 2. Non-contrastive learning (e.g., arxiv.org/abs/2102.06810, arxiv.org/abs/2110.04947) 3. Training dynamics in Transformers (e.g., arxiv.org/abs/2310.00535, arxiv.org/abs/2305.16380) 4. Grokking behaviors (arxiv.org/abs/2509.21519) 5. Spontaneous symmetry breaking (arxiv.org/abs/1703.00560) 6. Mechanism in forming symbolic solutions from gradient descent (arxiv.org/abs/2410.01779) I am the first/solo author in most of the works listed above. Code are here: github.com/yuandong-tian/…

@nico_anguita Thanks @nico_anguita!

@KuninDaniel looks great!

For the last few years, a lot of my work has been driven by the feeling that deep learning is not magic — there are principles, mechanisms, and laws waiting to be understood. This paper is our attempt to say that clearly!

Wir mussen wissen. Wir werden wissen

Very happy to be part of this project. We've compiled the main reasons why a Theory of Deep Learning is possible if not inevitable!

Great perspective on the theory of deep learning from a stellar group of authors!Physics-inspired ideas will play a central role in shaping this field. Congrats to my group alumni @blake__bordelon and @ABAtanasov for their contributions here and across many influential papers.

100% agree. Neuroscience embraces studying the brain at multiple levels — computational, algorithmic, and implementational. I’m excited to see deep learning moving toward the same conversation, with theory and interpretability informing each other!

Great to see the next generation taking the lead in the science of deep learning! Also proud that two brilliant members/alumni of my group are a part of this: @KuninDaniel & @MasonKamb

@SuryaGanguli @MasonKamb Many of the ideas, works, and intuitions discussed in Sections 2.1–2.5 grew out of your group — I’m very grateful to have been part of the Ganguli gang!