@JohnCLangford Updates on the Dion codebase (github.com/microsoft/dion), please check them out!
- Dion2 (arxiv.org/abs/2512.16928), which has much simpler math than Dion.
- NorMuon (arxiv.org/abs/2510.05491) thanks to @li_zichong.
English
Kwangjun Ahn
73 posts
Kwangjun Ahn
@KwangjunA
Researcher at NVIDIA // ex-Researcher at Microsoft, PhD from MIT EECS
Cambridge, MA Katılım Şubat 2020
317 Takip Edilen718 Takipçiler
@KwangjunA and Noah Amsel came up with Dion2 arxiv.org/pdf/2512.16928 , a radical computational improvement on Dion which builds on insights from Trion openreview.net/pdf/1d31ed870f… about how to effectively do a scalable orthonormal update in the Muon style.
English
Kwangjun Ahn retweetledi
sacled up to 12.7B dense, 5.5T tokens.
- polynorm (optimized kernel)
- grouped diff attn (their work)
- parallel muonclip (adopt alltoall like mainhorse, essential, dion)
- 80M batch
it's still non-reasoning, also not moe though... keep pushing guys!
arxiv.org/abs/2511.07464
Seunghyun Seo@SeunghyunSEO7
they also dropped fsdp2 optimized muon. though they don't use muon for 2.6b dense model, i think it's just beginning and they are preparing larger one. they pipeline muon's comm-comp with calc flops and the code is neat. not sure if it's existing method. huggingface.co/Motif-Technolo…
English
@KwangjunA This seems to be similar to our recently shared work, where we use Discrete Cosine Transform (DCT) to perform a cheap low-rank projection of the momentum buffer, followed by Newton-Schulz orthogonalization. Check out our paper here: arxiv.org/abs/2505.17967
English
New improvement in Dion leads to a speedup that makes orthonormal updates (eg. Muon) more scalable for larger matrices. The trick: carefully using Newton-Schulz (on smaller matrices) as Dion's backend. Updates to our microsoft/dion codebase are coming soon---stay tuned!
English
Kwangjun Ahn retweetledi
Join us on Sept 24 at 8 AM PT for Microsoft Research Forum Season 2 – a virtual series highlighting purposeful research and its real-world impact, from fundamental exploration to advancing AI responsibly, scaling innovation through products and open source, and driving positive change for society. Register now: msft.it/6011scy27
English
Kwangjun Ahn retweetledi
@jxbz love the repo! clean code, good practices but still not overly over-engineered, triton kernels, well documented, simple reference implementations alongside optimized code. nice
English
Kwangjun Ahn retweetledi
English
@KwangjunA @jxbz @JohnCLangford @GagMagakyan It's super helpful + the repo have a lot of details!! thanks 🙏
English
Kwangjun Ahn retweetledi
I had wondered why there was no official Dion implementation by the authors... I guess now we know. This repository looks dope: FSDP Muon and Dion implementations, triton kernels for Newton-Schulz, and lots of practical advice
(1/2)
English
@jxbz @JohnCLangford @GagMagakyan Thank you for the kind words! Hope this proves useful to the community!
English
Kwangjun Ahn retweetledi
Looks like extremely exciting and useful work by @KwangjunA, Byron Xu, Natalie Abreu, @JohnCLangford and @GagMagakyan
github.com/microsoft/dion/
(2/2)
English
Kwangjun Ahn retweetledi
[1/6] Curious about Muon, but not sure where to start? I wrote a 3-part blog series called “Understanding Muon” designed to get you up to speed—with The Matrix references, annotated source code, and thoughts on where Muon might be going.
English
Kwangjun Ahn retweetledi
Apparently Dion is now being worked on for Torch Titan: github.com/pytorch/torcht… :-)
Mikhail Parakhin@MParakhin
Since nobody asked :-), here is my list of papers not to be missed from ICML: 1) Dion: distributed orthonormalized updates (well, technically not at ICML, but everyone's talking about it). 2) MARS: Unleashing the Power of Variance Reduction for Training Large Models 3) ...
English
Kwangjun Ahn retweetledi
Since nobody asked :-), here is my list of papers not to be missed from ICML:
1) Dion: distributed orthonormalized updates (well, technically not at ICML, but everyone's talking about it).
2) MARS: Unleashing the Power of Variance Reduction for Training Large Models
3) ...
English
@orvieto_antonio @micahgoldblum @teodorasrec @jonasgeiping Nice results! One question: wouldn’t large (global-)batch size be more practical for distributed training? Does that mean still SGD is not effective for large scale?
English
As @micahgoldblum and coauthors, we also found that small batches make SGD effective in LM training. It's cool that our papers came out around the same time, and each has a different perspective!
Below, our take on why this happens.
Our awesome team: @teodorasrec @jonasgeiping
Papers:
arxiv.org/pdf/2506.12543
arxiv.org/pdf/2507.07101
English
Kwangjun Ahn retweetledi
But actually this is the og way of doing it and should stop by E-2103 to see @jxbz and Laker Newhouse whiteboard the whole paper.
Jeremy Bernstein@jxbz
Laker and I are presenting this work in an hour at ICML poster E-2103. It’s on a theoretical framework and language (modula) for optimizers that are fast (like Shampoo) and scalable (like muP). You can think of modula as Muon extended to general layer types and network topologies
English
Kwangjun Ahn retweetledi
Schedule-Free methods, which forgo cosine/linear schedulers by averaging iterates and computing gradients at interpolated points, yield smoother training curves. It's still unclear why they work well, and this paper explains the phenomenon through the river-valley loss landscape.
English