Sabitlenmiş Tweet
CM
862 posts
CM
@Creative_Math_
Deep Learning Research intern @blocks, grad student @UofT 🇨🇦 in a cool lab. Did pure math in a past life. cashmere-y
Toronto, Ontario Katılım Eylül 2020
350 Takip Edilen3.1K Takipçiler
I've been using GLM5.2 as a daily coding driver for ~2 weeks now
1) It doesn't suggest/catch good/bad ideas pertaining to ML like gpt5.5 or opus4.8 can
2) It's especially bad at SLURM
3) It's good enough for simple "just do it" stuff and you'll get it at 150+ tps
In conclusion I use it for most implementations where I'm dictating what to do but refer to 5.6 sol / Fable 5 for things I'd consider hard
English
“AI writes better code than I do” is a strange thing to say when you think about it.
AI was trained on a vast corpus of published code. It reflects the average of all that code.
So what you’re really saying is:
“My code is below average.”
English
On Aurora: before trying and noting “I didn’t see gains”, first see the distribution of your row norms in the update matrix itself
Often, you don’t really see neuron death at all (Normuon does a good enough job) so it doesn’t help cuz it’s solving a problem you aren’t facing
English
CM retweetledi
there are a lot of benchmarks that suggest 5.6 sol is the best model in the world right now, but the most reliable way to tell is that elon is obsessed with me again
English
I reject RMS matching as a whole, it changes the geometry and leads to attention logit instability even in small models (which means sth is def wrong there). I’m not convinced apriori that the update RMS for 2 diff optimizers should be the same, esp not when it’s muon (which has a specific geometry)
English
@Creative_Math_ Bonus: DeepSeek's Nesterov Muon also does "RMS-matching", but with a factor of 0.18 instead of 0.2 😂
x.com/EIFY/status/20…
EIFY@EIFY
DeepSeek opted to train DeepSeek V4 with Nesterov Muon. Nesterov Muon also has a momentum-dependent eff. LR albeit more complicated. Their "RMS matching factor" is 0.18: ((1 + mu)/(1 - mu)) ** .5 * (1 + 2 * mu - 2 * mu ** 3) ** -.5 * 0.18 = 1.03 if mu = 0.95 2/3
English
@navvye @ArdaGoreci I read his blog and agree the muP scaling is the right way to go, it’s just that NorMuon/some kind of row normalized version of Muon is significantly better than just Muon, and it’s still a bit unclear what’s best to do there
CM@Creative_Math_
btw, if you're using Muon w muP scaling, you don't need QK-norm/MuonClip to stabilize attention logits. Under muP scaling ||ΔW|| = 1 per-step so |Δ (q_i . k_j)| <= d and empirically logits stay stable too You prolly still want QK-clip just to be safe but it'll rarely activate
English
Shenyang Deng ✈️ ICML2026@DengShenyang24
NanoGPT is a good benchmark in the few-token regime, but its results might not generalize to the rich-token regime. Optimizer behavior may reverse in the mid-to-late stages. Similar observation was shared in @wen_kaiyue prior benchmarking work: arxiv.org/abs/2509.02046
QME
@SmartPig_Joe yeah imo i see nanogpt speedrun as a pool of ideas that might be interesting to scale
English
I am cautious about the optimizer built on modded nanoGPT Track3. Its arXiv paper lacks training on larger datasets and longer token horizon. Without clear scaling curves and steady acceleration ratios, Aurora’s edge cannot be proven scalable.
Tilde@tilderesearch
~1/7~ Today we are releasing Aurora on Arxiv → a leverage-aware spectral optimizer built for the MLP matrices in language models. Aurora achieves strong pre-training efficiency gains, achieves state-of-the-art performance on the optimizer track of modded nanoGPT speedrun among spectral optimizers, and allows for the effective training of very wide MLPs. 🧵
English
I’ve had the pleasure of chatting with @mandylu and get her big brain insight about how she would learn ai in 2026
what’s great with mandy is she did a whole lot of work in the industry (google, apple, etc) and did hella lot of research
so her advice is as informed as it gets
English
CM retweetledi
Damn, I almost switched to OpenCode but they don't do it either
There is a fix though, codex is open-source, so you can get an agent to just edit Codex to set the openAI API compatible endpoints as separate models in Codex, so you get the best of both worlds
CM@Creative_Math_
Anthropic, please release a GUI where you can render LaTeX and view diffs like Codex app does and I promise you have my loyalty forever
English
