Fabian Schaipp

Learning rate schedules seem mysterious? Turns out that their behaviour can be described with a bound from *convex, nonsmooth* optimization. Short thread on our latest paper 🚇 arxiv.org/abs/2501.18965

very nice paper finding that Muon & SOAP improve training of models for molecular simulations #ICML2026

Poster today at 5pm! #ICML2026

🎬 Field Notes: is more data always better? Physical AI is booming, and so are the data providers feeding it. But what actually makes a robot smarter? Awesome to chat with @fabinsch1, researcher at @inria who just presented his thesis in policy learning for robotics. 🧵

I will be at ICML from Wednesday, happy to chat on this! 🇰🇷

Why would you want to do this? 1) With our law, you get the correct scaling for the optimal batch size with two/three batch sizes per sweep 2) You can also derive scaling laws for *suboptimal* batch size intervals (e.g. which batch sizes waste at most 5% of compute)

How to scale the batch size in LLM pretraining? New paper on a scaling law that splits token budget D into training steps and batch size. 💺 arxiv.org/abs/2607.01487

@cjmaddison @orvieto_antonio :+1:

@FSchaipp @orvieto_antonio Thanks! The batch term is functionally a variance term, is that right?

Hot question: how should we allocate tokens: more steps or bigger batches? How big is "too big" for our batches? @FSchaipp proposes to fit a simple **three**-term law: L(N,M,K)= E + A·N⁻ᵅ + B·M⁻ᵝ + C·K⁻ᵞ, where M is the batch size in tokens and K is the number of training steps. Compared to works that fit batch-size laws directly, here batch size enters the loss law itself, so optimal and critical batch sizes are derived rather than being fit separately. Since D = MK (tokens), the law predicts an optimal batch size M* ∝ D^(γ/(β + γ)). paper: arxiv.org/pdf/2607.01487

@cjmaddison @orvieto_antonio will have a look. the batch size and steps terms are essentially resembling the structure of convergence bounds (details in the paper)

@orvieto_antonio @FSchaipp I wonder if these loss models are related in some way. Should be easy to figure out.

@_Hiroki11x yes yes, and I am surprised how many papers get this wrong! you also might enjoy our work on schedules, which already did the shape/gauge separation: arxiv.org/abs/2501.18965

Finding 1: the base learning rate matters far more than which schedule family you choose. Takeaway: tune the base LR first, and re-tune it for every shape. Comparing shapes without doing this is how you get misleading results. (4/7)

@giffmana @evaninwords wasn't meant seriously (sry it's 40° here)

@FSchaipp @evaninwords I'm not sure i follow why you say this tbh

This is the natural continuation of this group's previous several works on optimization, and I really like their style. A lot of experiments to look at all possible details. As opposed to a wall of theory and then one single experiment with untuned baselines.

awesome blog post! the Apertus project keeps on pushing optimization research for open LLMs.

@iamgrigorev wdym with 1.5x compute? spent on what?

EMA is such a strong technique for pre-training. You can spend 1.5x more compute and get 0.01 eval loss improvement or just apply EMA at final stages of cooldown and consistently get the same result.

Judging optimizer gaps by looking only at language modeling with a fixed batch size is dangerous: one gets only 1/2 of the story. @orientino_ and @ruuustem_10 went beyond. Turns out that for every model, task, and data, there is always a setup where Adam > SGD. 🧵

@DimitrisPapail ah, bsz is probably fixed in the benchmark so nevermind

@DimitrisPapail I'd try small batch size for sgd

taught grandpa SGD to speedrun GPT tonight 2x slower than Muon. We are 0.10 nats from god and $300 of runpod credits poorer... yet doing the single most dangerous thing I can: leaving the agents (codex and CC with fumble) run overnight 🫡 see wassup tomorrow. either 3.28 at 550s or go home with 300$ less

@ruuustem_10 @orvieto_antonio @n_ajroldi yeah, agree! i guess you can get sth similar for the smooth case too, but probably need to adapt the alpha< 2/L step

@FSchaipp Great work! With @orvieto_antonio @n_ajroldi we most of the times observed that NGN step size is more stable than SPS. Interesting to see that it’s also somehow suggested by your theory. Can the same be done for convex-smooth losses?

"It's easier to tune the LR for method A than for B." We tried to formalize this for model-based stochastic optimization methods. We find a key quantity, called stability index, that describes how stable a (weakly) convex bound is as a function of LR. 📚arxiv.org/abs/2602.09842

The theory is with SGD as base update step. But all these adaptive step sizes can be used to obtain practical methods. For example, Polyak step sizes in combination with - Muon (@CrichaelMawshaw): arxiv.org/abs/2510.09827 - ScheduleFree (@aaron_defazio ): arxiv.org/abs/2605.19095

Joint work with @gowerrobert and @TaylorAdrien , and just accepted at #ICML2026.