Wu Lin

113 posts

Wu Lin

@LinYorker

Postdoctoral fellow at @VectorInst. ML PhD at UBC. Mathematical and computational structures for ML. Geometric and algebraic methods.

Toronto, Canada Katılım Ağustos 2020

40 Takip Edilen317 Takipçiler

Sabitlenmiş Tweet

Wu Lin@LinYorker·16 Tem

#ICML2024 Can We Remove the Square-Root in Adaptive Methods? arxiv.org/abs/2402.03496 Root-free (RF) methods are better on CNNs and competitive on Transformers compared to root-based methods (AdamW) Removing the root makes matrix methods faster: Root-free Shampoo in BFloat16 /1

English

12.9K

Wu Lin@LinYorker·12 Haz

@MarktHart125849 I mean this repo. github.com/kellerjordan/m…

English

Mark 't Hart@MarktHart125849·12 Haz

@LinYorker If you looked at it, you'd know it's a vanilla Transformer and has only been going on for 1.5 months

English

Wu Lin@LinYorker·12 Haz

On one hand, it is essential to tune baseline methods well on a model. On the other hand, it may be better to avoid using a model/architecture that has been modified and optimized for a single method for 1.5 years.

Konstantin Mishchenko@konstmish

I just submitted a PR to modded-nanogpt with better hyperparams. With them, Muon can reach the target loss after 3250 steps instead of 3325. Always tune your baseline well when doing research. Weak baselines can make any idea look promising

English

1.2K

Wu Lin@LinYorker·28 May

Some initial steps to make Shampoo and SOAP faster arxiv.org/abs/2605.26327 We are working on further improvements.

English

Wu Lin@LinYorker·3 May

We will make Shampoo/SOAP, including KL-Shampoo/KL-SOAP, faster. Our goal is to match Muon's runtime while maintaining Shampoo/SOAP's strong per-step performance. Stay tuned for new updates.

Lucas Nestler@Clashluke

KL Shampoo and KL SOAP outperform their non-KL counterparts by learning the preconditioners compositionally, so that each stage corrects what remains after the last. Available in HeavyBall 3.1.1, with major PSGD stability backports.

English

3.1K

Wu Lin@LinYorker·28 May

@wen_kaiyue You may want to have a look at this paper for further improvement arxiv.org/abs/2605.26327

English

159

Kaiyue Wen@wen_kaiyue·27 May

Soap with preconditioning frequency 1 is a strong optimizer!

Keller Jordan@kellerjordan0

Modded-NanoGPT optimization result #19 (2026/05/08): @wen_kaiyue has achieved 3125 steps using KLSOAP-Hyperball, matching the record.

English

8.5K

Wu Lin@LinYorker·1 May

ZXX

Wu Lin@LinYorker·1 May

also, short sided KL-Shampoo = short-sided Shampoo^2 = Muon

English

114

Wu Lin@LinYorker·1 May

@_arohan_ Muon

Tiếng Việt

240

Wu Lin@LinYorker·19 Nis

@weijie444 Looks like a KFAC-based method with modern clipping? G(ZZ^T)^{-1} is known as the FOOF update arxiv.org/abs/2201.12250 while msgn() can be interpreted as "generalized (preconditioned) gradient norm clipping" arxiv.org/abs/2506.01913 .

Weijie Su@weijie444

We released "The Newton--Muon Optimizer" . We show that Muon is secretly an implicit Newton method, and use this insight to build a better one. 1/n Paper: arxiv.org/abs/2604.01472

English

5.8K

Wu Lin@LinYorker·3 Ara

@MarkSchmidtUBC @EmtiyazKhan This work is a joint effort with Scott C. Lowe, @f_dangel, @runame_, Zikun Xu, and @RogerGrosse. Stay tuned for more updates coming soon.

English

115

Wu Lin@LinYorker·3 Ara

This work builds on my ICML 2019 paper (with @MarkSchmidtUBC and @EmtiyazKhan), extending a variational Bayes-based geometric framework to modern NN optimization. It can be used to design methods for Bayesian inference, numerical optimization, and gradient-free optimization.

English

126

Wu Lin@LinYorker·3 Ara

Within an information-geometric framework, we reconnect Shampoo/SOAP with both classical quasi-Newton ideas and Gaussian whitening, and develop practical methods that naturally handle tensor-valued weights in language model pre-training. arxiv.org/abs/2509.03378 opt-ml workshop

English

1.1K

Wu Lin retweetledi

Runa Eschenhagen@runame_·20 Kas

1/9 In practice, the Shampoo optimizer crucially relies on several heuristics. In our NeurIPS 2025 spotlight paper, we investigate the role of learning rate grafting and infrequent preconditioner updates in Shampoo by decomposing its preconditioner. arxiv.org/abs/2506.03595

English

13.3K

Keşfet

@MarktHart125849 @wen_kaiyue @_arohan_ @weijie444 @MarkSchmidtUBC @EmtiyazKhan @f_dangel @runame_