Lizhang Chen

@Yuchenj_UW cannot agree more!!! x.com/lzchen_ut/stat…

Spud and Mythos are a reminder that pretraining still matters, a lot. RL is the cherry, not the cake.

Pre-training is definitely not dead--there are still lots of promising directions worth exploring, and it’s exciting to see Slowrun pushing on so many of them. thanks, @industriaalist.

1/ nanogpt slowrun 🐢 update: we're focusing on occasional big data efficiency updates, but we had a lot of interesting additions in the last few weeks, here's the rundown: · multi-token prediction (@clark_kev) · looped transformers (@cs_serdar) · test-time training (TTT) (@Sam_Acqua) · stochastic logits averaging (@bishmdl76, @ShmuelBerman) · stochastic depth (@ChinmayKak) · IHA attention (@madhavsinghal_) · tuning for gradients norm (@zhiweixux) · probability averaging + scaling ensembles (@lzchen_ut) · MuonEq-R (@clark_kev)

Excited to share that 3 papers will be presented at ICLR 2026: 📌 Cautious Weight Decay arxiv.org/abs/2510.12402 📌 Cautious Optimizers arxiv.org/abs/2411.16085 📌 DeMo: Decoupled Momentum Optimization arxiv.org/abs/2411.19870 Looking forward to presenting these works at #ICLR2026.

@goodhunt @fujikanaeda @goodhunt x.com/lzchen_ut/stat… github.com/L-z-Chen/data-…

more people should be talking about this github.com/NVIDIA-NeMo/Da…

@Yuchenj_UW I would ask my agent which one I should pick.

If you had two software engineering offers: > One pays you $500k/year salary, but covers zero LLM tokens. > One pays you $400k/year salary, but gives you $500/day free LLM tokens. Which one are you taking?

It really feels like Opus 4.6 has gotten dumber. @claudeai

It really feels like Opus 4.6 has gotten dumber.

Blog Post - Lion-K CCWD: Corrected Cautious Weight Decay and Hyperparameter Transfer Derivation of Lion-K with Corrected Cautious Weight Decay (CCWD) and transformation rules for hyperparameter transfer fixing Complete(d)P momentum

🔥🔥🔥

@claudeai API Error: 500 {"type":"error","error":{"type":"api_error","message":"Internal server error"},"request_id":"req_011CZ9Fwi7zGuT7Sfkg5ipRs"} @claudeai

@claudeai API Error: 500 {"type":"error","error":{"type":"api_error","message":"Internal server error"},"request_id":"req_011CZ9EfzB5ByTWKVenQ8trK"}

@claudeai again

🧠 Moving from compute-bound to data-bound! 🚀 Data-Run is a new open-source competition testing data efficiency in language model training. The goal? Hit a validation loss of <= 3.2 with the absolute minimum number of unique tokens. 📊 Curate, distill, or synthesize your best dataset and climb the leaderboard. 🏆 How few tokens do you need? 🏆🏆🏆 Show us what you've got! 👇 github.com/L-z-Chen/data-…