Wu Lin

#ICML2024 Can We Remove the Square-Root in Adaptive Methods? arxiv.org/abs/2402.03496 Root-free (RF) methods are better on CNNs and competitive on Transformers compared to root-based methods (AdamW) Removing the root makes matrix methods faster: Root-free Shampoo in BFloat16 /1

@weijie444 Looks like a KFAC-based method with modern clipping? G(ZZ^T)^{-1} is known as the FOOF update arxiv.org/abs/2201.12250 while msgn() can be interpreted as "generalized (preconditioned) gradient norm clipping" arxiv.org/abs/2506.01913 .

@MarkSchmidtUBC @EmtiyazKhan This work is a joint effort with Scott C. Lowe, @f_dangel, @runame_, Zikun Xu, and @RogerGrosse. Stay tuned for more updates coming soon.

This work builds on my ICML 2019 paper (with @MarkSchmidtUBC and @EmtiyazKhan), extending a variational Bayes-based geometric framework to modern NN optimization. It can be used to design methods for Bayesian inference, numerical optimization, and gradient-free optimization.

Within an information-geometric framework, we reconnect Shampoo/SOAP with both classical quasi-Newton ideas and Gaussian whitening, and develop practical methods that naturally handle tensor-valued weights in language model pre-training. arxiv.org/abs/2509.03378 opt-ml workshop

1/9 In practice, the Shampoo optimizer crucially relies on several heuristics. In our NeurIPS 2025 spotlight paper, we investigate the role of learning rate grafting and infrequent preconditioner updates in Shampoo by decomposing its preconditioner. arxiv.org/abs/2506.03595

Huge thanks to my amazing collaborators. This project was led by @juhanbae along with @LinYorker and @RogerGrosse. Supported (indirectly) by @Anthropic, @NVIDIA, @VectorInst, @UofTCompSci/@UofTArtSci/@UofT

Are you LoRA fine-tuning LLMs and looking for easy ways to get improvements in accuracy? And also Bayesian uncertainty on top for free? Then check our recent work, accepted @neurips24fitml workshop! arxiv.org/abs/2411.04421

Natural gradient descent: (steepest) gradient descent under a norm induced by the Fisher matrix yorkerlin.github.io/posts/2021/10/… Riemannian gradient descent (with geodesic retraction) : gradient descent in Riemannian normal coordinate

Some hardcore theory people complain that "second-order" methods in DL do not have a superlinear convergence rate. At the same time, they are happy to consider SGD a first-order method with only a sublinear rate.

@typedfemale @F_Vaggi @j_mcgraph FYI, many assumptions for convergence analysis may not hold in practice, according to this paper: arxiv.org/abs/2407.01825.

@typedfemale @F_Vaggi @j_mcgraph Like LBFGS, many second-order methods in DL are indeed quasi-Newton methods.

i've noticed many (including myself) say "2nd second-order" in machine learning to refer to a set of optimizers (FOOF, shampoo) that... don't actually use second-order information? idk it's weird

My talk on spectral transformers, given at Princeton workshop on learning in dynamical systems, is now online: youtube.com/watch?v=D_NwH5…

Lin (1991) definition of Jensen-Shannon divergence JS(p,q)= (KL(p:(p+q)/2)+KL(q:(p+q)/2))/2 is a *variational divergence* defined by JS(p,q)=min_{c} (KL(p:c)+KL(q:c))/2 where optimum value is c=(p+q)/2 Defined as information radius by Sibson (1969) 👉mdpi.com/1099-4300/23/4…

Looking forward to making non-diagonal methods faster and simpler.

We can use NGD to obtain many methods: natural evolution strategies (CMA-ES, NES), natural policy gradients, natural-gradient variational inference, exact Bayesian inference on conjugate models, Newton's method, root-free adaptive methods, and Riemannian GD on submanifolds.