mikail

This paper got into @Nature!!! 🚀🚀🚀 Look at @SarthakChandra’s thread for a summary x.com/SarthakChandra…

We now have four MoE models of roughly similar size, but based on very different architectural choices: >Qwen3 30B: full attention with simple GQA >Qwen3.5 35B: full attention + Gated DeltaNet >GLM 4.7 Flash: MLA >Nemotron 3 Nano: full attention + Mamba with very different KV-cache footprints. Nemotron is by far the most memory-efficient of the four. Although GLM 4.7 Flash uses MLA, its rank is fairly high, higher than Mistral Small 4 for instance, so it delivers only limited memory savings. To make these comparisons easier, I wrote an article and a notebook that estimate the memory consumption of all these models for any given sequence length: kaitchup.substack.com/p/the-kv-cache…

The hottest summer I ever spent was a winter in San Francisco

Or their complete optimizer setup. These all interact. Even the kernels and precision can varying effects on the training dynamics.

The blueprint for this "grand unification" already exists: --------- 🔹1961: Landauer established the thermodynamic cost of bit erasure (ieeexplore.ieee.org/document/53924…) 🔹1982: Bennett resolved Maxwell’s Demon, proving that logical irreversibility (erasure), rather than measurement, necessitates thermodynamic work (link.springer.com/article/10.100…) 🔹Late 1990s: Jarzynski and @gavincrooks related non-equilibrium work to equilibrium free energy. Their Fluctuation Theorems were later used to derive the entropy production cost of statistical inference (journals.aps.org/prl/abstract/1…; journals.aps.org/pre/abstract/1…) 🔹2007: Kawai et al. proved average thermodynamic entropy production equals the KL divergence between forward and backward path distributions( journals.aps.org/prl/abstract/1…) 🔹2009: Sagawa & Ueda incorporated information theory into this framework, bounding the energetic cost of measurement and feedback (journals.aps.org/prl/abstract/1…) 🔹2019: Wolpert extended stochastic thermo to formal computational architectures, including Turing machines and boolean circuits (iopscience.iop.org/article/10.108…) 🔹...and many more works in thermo/non-equilibrium stat mech that are waiting to be formally connected to computer science/ML concepts. --------- Formalizing this mathematical connection is a central research interest of mine. Reach out if you have ideas (DMs open).

Last year I have thought about two ideas about improving Residual Streams, one is RNN view, another is Attention view. The first one we called Deep Delta Learning (DDL, arxiv.org/abs/2601.00417), and the second one we called Transformer^2. Concurrently, Kimi released the Attention Residuals. Need to point out is that DDL is more friendly to Pipeline Parallelism for large scale models. Transformer^2/Attention Residuals need to transfer a lot of previous layers’ hidden states. @elonmusk DDL is definitely worth reading!

@ZimingLiu11 This is blowing my mind. Do you have intuition? Does longer sequence dilute signal that makes the gradient and hence any gradient based optimizer worse?

When do "Neural thickets" / RandOpt work? In today's blog, I show that sequence length is a key parameter -- RandOpt works better for longer sequences, while gradient-based methods work better for shorter sequences. kindxiaoming.github.io/blog/2026/rand…

Now comes the unusual bits: after enough hyperparams & other kinds of config sweeps, I’ve reached the surprising conclusion that no form of regular curriculum training beats _reverse_ curriculum training on this specific task! I.e. instead of starting with easy sudoku puzzles data and gradually ramping up toward harder ones, the training runs that start with harder puzzles and end with easy puzzles always fare better, and also fare better than mixed difficulty sampling It’s possible to justify this in hindsight; a friend mentioned “I naively thought I’d get better at chess by starting with Blitz (easy, fast) instead of regular chess (harder, slower), but it’s the other way around”. That seemed philosophically interesting. Experts: please check the training runs in case there’s been a mistake; but so far the numbers do seem to speak for themselves

Neural cellular automata are the retinal waves for LLMs? 🤔

1/ Dense MLPs are a lie. The standard transformers we train are already doing sparse routing inside their feedforward layers—we just couldn't see it until now. 🧵

My dear young person, Don’t succumb to mediocrity. There’s enough of it going around. Aspire for craftsmanship, as that is what leads to joy and beauty. The world needs more people who’re proud of what they make, and less of those who couldn’t care less.

"Taste transfers even as domain knowledge shifts. Theoretical physicists become quants. Quants move into AI." 🤔

Does anyone know ballpark numbers for typical condition numbers of gradient matrices during training

What if we could mathematically predict how a neural network evolves during training? We developed the first mathematical framework that explains why trained networks develop the distinctive "bulk+tail" weight structure that predicts generalization, validated across transformers, vision transformers, and MLPs. 📄 arxiv.org/abs/2507.12709 1/7

My thesis is now accessible online! I've tried to make it the introduction to non-Euclidean methods and monotone operators that I wish I had when starting out. 1/n infoscience.epfl.ch/entities/publi…

Striking image from the new Anthropic labor market impact report.

Muon can accelerate LLM training, but does that benefit transfer to regulatory DNA sequence modeling with its different data distribution? 🧬 Our results show that Muon with independent weight decay (MuonW) hits our validation perplexity target in ~37% fewer FLOPs than the best Adam configuration.

@SynBio1 A full 4 year turnaround time

@SynBio1 My Nature paper was submitted when GPT-3 was out and accepted when o1 was out 🫠🫠🫠🫠 nature.com/articles/s4158…

It took Nature 13 months to publish Evo 2! 13 months! For reference: Opus 4.5 ended software engineering as we know it 4 months ago. ClawdBot added 2 million users in a single week in January. Academic publishing is so cooked it's not even funny nature.com/articles/s4158…

@CalcCon 👀

New results coming soon showing how SETOL (the theory behind weightwatcher) can be used to merge models