rohan anil

It turns out multi step backpropaganda is better. paper has a beautiful way of improving backpropagation. One iteration cleanly gets us backprop, multiple iterations get us a preconditioned update.

@LakshyAAAgrawal @agarwl_ Very cool! Thanks, Nice work!

Figure 14 (in appendix) provides a breakdown of various ablations and comparative experiments. First, we see that FST-trained model, even without the co-optimized prompt (“Slow only (FST w/o prompt)”, green) gains +4.2pp over RL-only model. Next, FST with the co-optimized prompt (“Slow + Fast (FST)”, green) outperforms RL-only model with GEPA-optimized prompt (“Slow + Fast (RL + GEPA)”) by +3.8pp, which increases further if we run GEPA again on FST-optimized model instead of just taking the co-optimized prompt. Further, we see this on other datasets as well: ⦁ On HoVer: FST 30.8% vs RL+GEPA 24.9% (+5.9pp), even using 3x fewer data points ⦁ In easy-to-hard generalization, training on Polaris and testing on HMMT25, we see: FST 39.7% vs RL+GEPA 37.1% (+2.6pp), using 1.4x fewer datapoints. This is with the co-optimized prompt. Running GEPA on top of the FST model achieves 41.4% (+4.3pp).

Training LLMs is synonymous with updating their weights. However, LLMs can also learn in-context using *frozen* weights. There is no good reason for restricting learning to being in-context or in-weights. So a natural idea is "Learning, Fast and Slow" (FST). In FST, slow learning is LLM weights trained with RL while fast learning is context / prompt (fast weights) optimized with GEPA. Compared to RL, FST performs better while being more data efficient, adaptable (plasticity), and forgetting less (stays closer to base models). I think this idea of learning both fast-slow weights would be a good foundation for continual learning. PS: Geoff Hinton (the OG) described the idea of fast weights and slow weights several years ago, and back then I remember thinking it's a very cool idea. See more details here: gepa-ai.github.io/gepa/blog/2026…

@giffmana I forgot to do laundry, had to pull this out

@_arohan_ I'm scared of the day mine gets worn out

Collectors edition

Decoupled and Dualized with a side of fries

@DakshMalik47 @CoreAutoAI Conspiracy theory - approved by Jt

@_arohan_ @CoreAutoAI i bet u read this 1st on slack , then Jerry approved to post this and here u r trying to get more likes than the original post. i understand the game Rohan

We have an agent that looks like a dog with a surfboard and thats making me uneasy

@typedfemale At least Günther Schulz gave credit to Newton.

@_arohan_ i have a chrome extension that replaces "muon" with "shampoo"

they just created a million muon variants

@typedfemale It was inevitable with such a good name

@typedfemale #practical-shampoo

golden age for optimizers right now. every day you see another SoapyShampooGluon ^-1/2 (RMSMatched) drops

@tonysilveti @dakovalev1 Nice!

@_arohan_ and DASGO for the instantaneous version; now I remember this paper by @dakovalev1 who made this connection explicit too (with analysis)

Wow! More theoretical analysis linking the spectral norm and row-norm. They make a nice argument using "row-block diagonal dominance" of the layer-wise Hessian to say that spectral LMO and row-norm LMO should give equivalent asymptotic dynamics (as width grows).

Uh oh. Captain, its just thursday

Neural networks do math by rotating shapes. We found a shape-rotating calculator hidden inside an LLM – and it’s used for more than just math! (1/6)

@DengShenyang24 @tonysilveti Aurora isn’t using diagonalizations at all.

This conclusion isn't unique to our work. In fact, a number of recent concurrent works have arrived at the same finding: [7] A Minimalist Optimizer Design for LLM Pretraining. [8] SRON: State-Free LLM Training via Row-Wise Gradient Normalization. [9] Mano: Restriking Manifold Optimization for LLM Training. [10] On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer. [13] Aurora: A Leverage-Aware Optimizer for Rectangular Matrices.

@DengShenyang24 @tonysilveti Are you saying this diagonalization matches non diagonal update? Highly doubt it, and think your baselines are weak.

Prior work proposed fairly abstract frameworks and only adopted the diagonal-dominance approximation at implementation time, due to compute constraints. You can't say they understood why this trick works for neural networks. Common optimization-theory setups are not equivalent to the neural-network optimization problem. Nonconvex smooth optimization, OCO problem setups, and many overly abstract theoretical frameworks simply cannot explain why various tricks work specifically on neural networks. You can read our blog to understand why prior optimization-theory work fails to predict Muon's success.

@tonysilveti It was called one sided in code :) github.com/google-researc…

@_arohan_ Yes, I also found it now here and indeed the name was "simple" rather than one-sided.

@tonysilveti arxiv.org/pdf/2002.09018 Also in Appendix of distributed shampoo paper

@_arohan_ ...I didn't see that connection before. Yes, if you take as convention that one-sided shampoo changes the power from -1/4 to -1/2 (which makes sense). But I don't find one-sided shampoo mentioned in either the original or your followup paper. Is there a different name used?

@t_cmtl @tonysilveti arxiv.org/abs/1901.11150 (search on Google :P )

@_arohan_ @tonysilveti What is your SM3 work? Could you kindly share it?