Thomas Pethick

1/ We introduce SODA: a simple optimizer wrapper that improves a base optimizer, adds no hyperparameters, and removes the need to tune weight decay. The wrapper provides consistent improvement. Most notably, SODA(Muon) beats Muon even when Muon gets a tuned weight decay sweep.

@tonysilveti I got a bit delayed but here we are: x.com/tmpethick/stat…

@tonysilveti Thank you for sharing! you beat me to it 😂 I'm really excited about this direction - will share some more thoughts tomorrow

SODA shows that Muon, Lion, NAdam, and others are all special cases of Optimistic Dual Averaging! A simple wrapper around any base optimizer can replace weight decay tuning with a principled 1/(k+2) schedule; no new HPs. Practical results seem promising! arxiv.org/abs/2605.11172

7/ Thanks to @CevherLIONS for supporting it and getting Roman Macháček on board, and @WanyunXie for scaling up the experiments and debugging runs together – it’s always a joy

6/ There are a lot of interesting questions one can ask from this perspective — please check out the paper! Paper: arxiv.org/pdf/2605.11172 Code: github.com/tmpethick/soda…

1/ We introduce SODA: a simple optimizer wrapper that improves a base optimizer, adds no hyperparameters, and removes the need to tune weight decay. The wrapper provides consistent improvement. Most notably, SODA(Muon) beats Muon even when Muon gets a tuned weight decay sweep.

@EIFY @tonysilveti One reason I find z0 appearing interesting is for finetuning where weight decay is otherwise typically not used (I added a comment on this in the conclusion) - but that’s a story in itself

@EIFY @tonysilveti We haven’t ablated this beyond 1 x chinchilla with a 124M model (the main setting where I tested things before we extrapolated across horizon and model size)

@CV_novel_plume With that said I think the ODA perspective is interesting in itself and there's more to extract

@CV_novel_plume Yes, this is exactly the SODA wrapper! I wanted to extract something concrete from the perspective and I was at the time trying to understand weight decay - surprisingly the first thing I tried (just using params from theory) worked without any retuning of lr etc of the base opt

Great work! I think the core idea is simple: replace tuned weight decay toward zero with an annealed pullback toward initialization. - Baseline WD shrinks weights toward 0 with a tuned hyperparameter (usually coupled w/ lr schedule). - SODA instead recenters weights toward x_0 at initialization with a fixed 1/(k+2) schedule.

@_arohan_ Yes, non-constant is interesting to explore. The GPA paper by @aaron_defazio has a very nice perspective on diloco also through schedule-free, which makes the delta/diff from SODA easier to understand (I've compared in the related work section)

This is very nice, diloco will likely find it useful for outer steps.

@wen_kaiyue @tonysilveti Your comment about optimism is interesting. I mainly focused on extracting a schedule for weight decay in this work, but there is an interesting question on how to schedule optimism and weight decay in tandem to better exploit smoothness

@wen_kaiyue @tonysilveti Its actually not quite batch size independent - if you squint the convergence theorem suggests a constant then 1/√k (the changepoint will depend on noise), but we didn't investigate this much empirically yet

@tonysilveti Yeah, except even for MLP blocks RMSNorm(W2σ(W1)) its ok for both matrices as long as the activation function is positively homogeneous (e.g., true for ReLU and ReLU^2)

Makes sense! Also gives a suggestions of which parameters it is okay to regularize as in MuonH and which you really ought to allow to grow in an unconstrained fashion. If they are not followed by some kind of layer/RMSnorm, then maybe don't do the Frobenius normalization?

Why is Frobenius weight normalization ok when combined with non-Euclidean steepest descent methods? A short note: pethick.dk/posts/2026-05-…

I wrote up a quick note here expanding a bit on it: pethick.dk/posts/2026-05-…

It is common to exclude the unembedding layer from using weight decay, but why? Essentially, applying weight decay to the last layer will constrain the logits of the model and consequently prevent high-confident output. 👇 x.com/tmpethick/stat…