Robert M. Gower @ Neurips 2025

Do you want to do a Postdoc developing new methods/theory in Optimization for deep learning/ML? Do you enjoy bluesky open research and discussions on black boards? Then Apply to the Flatiron Fellowship in the Center of Computational Mathematics simonsfoundation.org/flatiron/caree… 1/3

@bozavlado @giffmana Yeah, this is mad, and the same issue always of ADMM methods applied in this way. Unless these copies of models are distributed across different machines, in makes no sense!

Nobody talks about the fact that these optimizers actually need M times more memory (where M is the number of data chunks). Also: There is a huge advantage in wall-clock time on the XL model (comes from parallelization). Small advantage on loss vs number of tokens. No advantage on Nano. Maybe baselines for XL were taken from Nano.

New optimizer with earth-shattering plots making the rounds, and published in Nature too (Machine Intelligence, but let's just drop that part.) So of course I had to take a quick look. A few things I noticed that make me a bit sus, though I'm not saying to outright discard it - Each point is caption of the corresponding screenshot below: 1. What on earth are these SGDM vs AdamW gaps? They are not normal -> untuned baselines? (Also: what good is a Nature MI editor, if they approve plots with "0M" everywhere on x-axis???) 2. For vision models they tune lr's, good. But not wd or other optim hparams, meh/sus. 3. For LLM, they select hparam on test. At least epochs, but given this and that they seem to use "validation" and "testing" as synonyms in the paper, probably everything. 4. I am not sure a Medium blogpost tutorial with an arbitrary hparam selection is a good starting point for the baseline of a Nature MI paper?? Maybe this new optimizer is as amazing as promised, but I'll need to see less suspicious evidence. I wish the reviewers had asked for that. Maybe someone put it to test on the nanogpt speedrun? At least that has heavily-tuned baselines, including optimizers.

@mher_safaryan @LancasterUni Brilliant news, congrats on your new position!

Life update! I’m excited to share that I’ve started a new role as an Assistant Professor in the School of Mathematical Sciences at @LancasterUni. Our section MARS (Mathematics for AI in Real-world Systems) is currently recruiting PhD students and Senior Research Associates. 👇

@hayou_soufiane @FrancoisChauba1 @agupta Even some cs students only do theory and don’t use gpus

@FrancoisChauba1 @agupta It would make more sense to look at GPUs/(students who use GPUs), not all students (e.g. most social sciences students don't use GPUs). At Hopkins, that ratio is much higher and compute is very available currently.

Last night, @agupta and I hosted a great dinner with 14 professors at #NeurIPS2025 from leading academic labs across the US, and many cited compute in academia as "abhorrent". Out of curiosity I just pulled these stats. This is insane. To do meaningful AI research today you need at least 1 GPU/student. Likely 8+ to be honest. The best university (Princeton) is at 0.8 GPUs/student. Stanford is at 0.14 GPUs/student. Marlowe (Stanford's "super cluster") has only 248 H100s for the whole CS Dept to use. Every frontier lab has >100k. This needs to be fixed.

Check out my poster today (Thurs) at 11am--2pm session. Exhibit Hall C,D,E Poster Location: #602 "Fisher meets Feynman: score-based variational inference with a product of experts" (NeurIPS spotlight) with @gowerrobert David Blei and Lawrence Saul @FlatironInst #NeurIPS2025

We’re recruiting for both postdoc and open-rank positions. Learn more about ML@CCM 👉 users.flatironinstitute.org/~lsaul/ml_ccm.… I’ll also be in San Diego for NeurIPS — feel free to DM if you’re interested in #AIforScience or #GenerativeAI

@tonysilveti Oh right, I understand. We need the inequality to handle the zero case. True

@gowerrobert 0, which is not in the set of directions with norm = 1. If you take the inequality, you can define the LMO at 0 to just be 0.

We've just finished some work on improving the sensitivity of Muon to the learning rate, and exploring a lot of design choices. If you want to see how we did this, follow me ....1/x (Work lead by the amazing @CrichaelMawshaw)

@tonysilveti I don’t understand. If the input (gradient) is zero, what would be a good direction? Also, the inequality and equality constraint give the same solution when maximizing a linear function. Both solutions must be on the boundary (satisfy the equality constraint)

@gowerrobert The way you define the LMO (following old optimizer new norm) is not correct for the algorithm you want. If you take the minimization over the equality constraint, then you get a bad direction when the input is zero. The correct way is the frank-wolfe (Scion) way with inequality.

@bremen79 @roydanroy @2prime_PKU @YouJiacheng That sounds interesting, any paper reference in this other prox approximation ?

@roydanroy @2prime_PKU @YouJiacheng Sure, but I guess here it was more about "this idea cannot be that new" and in fact, from a certain point of view, it is not. A different approximation of proximal updates were present even in Vowpal Wabbit (MSR software), sooner or later people will rediscover that too

2019? so this truncation method is younger than Transformer?

@2prime_PKU @YouJiacheng I couldnt find anything about truncation or even polyak stepsizes in Nesterov’s or Bubecks (2015) book. Could you please share a screenshot of where you saw this?

@YouJiacheng This is polyak's stepsize proposed at 1963. The truncation view is already in textbooks e.g. Nesterov (2004), Ch. 2; Bubeck (2015), Sec. 3.3.

@fnnBsch @CrichaelMawshaw Ok I see, very interesting. Going to look into this some more, cuz tuning 2 lrs sucks. Thanks!

@gowerrobert @CrichaelMawshaw I think so, see p.4 in the report ("Muon can directly reuse the learning rate and weight decay tuned for AdamW") and Appendix A. It seems that the RMS relationship is quite straightforward. Chances are the LR setup you found to work best is in the same ballpark?

@fnnBsch @CrichaelMawshaw We tuned the two lrs of the Adam and muon layers separately for all methods. Does the moonshot scaling really allow for one shared lr for Adam and muon ? I didn’t know that, and that would be great if it’s true. It made a big difference tuning both lrs for us

@gowerrobert @CrichaelMawshaw Not sure if this was covered but it seems that the MuonAdam variant uses two different LRs. Have you also tried the Moonshot muon variant (arxiv.org/pdf/2502.16982) where Adam + Muon share LR?

@achandan47 @CrichaelMawshaw Good question, since we only change the norms, but always have the same inner product, I don’t think there would be any change to this Cautious weight decay

@gowerrobert @CrichaelMawshaw On a related note, what do you think about Cautious Weight Decay combination with your Muon variant? arxiv.org/pdf/2510.12402

@YouJiacheng @Jianlin_S I would be happy to collab on such a blogpost. Me and @CrichaelMawshaw are writing some intro lecture notes on an intro to non-Euclidean gradient descent. Also I’m a big fan of @Jianlin_S blog!

WOW that's VERY COOL!!! cc @Jianlin_S -- we need a blog on steepest descent tricks like truncation

Diana was the driving force behind all our variational inference work, and any department would be lucky to have her!

Here are some slides I made for a short tutorial on Non-Euclidean gradient descent and these variants: docs.google.com/presentation/d…

Our paper covers a lot ground, including exploring different product norms, formalizing MuonAdam as steepest descent, introducing the combination of truncation + Muon, and a lot of experiments! Here are the details -> arxiv.org/pdf/2510.09827 and ...