Dmitry (Dima) Lepikhin

At this scale (10T+ params), pre-training doesn't just average—model capacity explodes, letting rare signals carve out distinct subspaces in the latent space without dilution. Novel ideas in data (e.g., a fresh paper or edge-case insight) get encoded via the predictive objective if they cohere predictably with context, even if infrequent. Emergence kicks in: the model starts recombining latent patterns into outputs that feel "new" because no single training example had them exactly. It's not invention from void—it's hyper-efficient compression revealing unseen connections in the data distribution. Post-pretrain fine-tuning or prompting amplifies it further.

Today, we see the result: Muse Spark. I'll be honest, I was surprised how competitive this model is. Progress at OAI, Anthropic, and GDM has been continuous, built on compounding breakthroughs. But MSL's team seems to have taken a single binary leap, catching up on many fronts at once.

08 Champs ☘️

I painstakingly ran all 20 EsoLang-Bench hard problems through Claude webui. It solved 20/20 (100%). No specialized scaffolding, no expert prompting, no few-shot examples, it just solves them natively. This benchmark just suffocated the models with constrictive scaffolding.

Ohhh nooo not my private IP how dare someone use that to train an AI model, only Anthropic has the right to use everyone elses IP nooooo, this cannot stand!

We ran a randomized controlled trial to see if LLMs can help novices perform molecular biology in a wet-lab. The results: LLMs may help in some aspects, but we found no significant increase at the core tasks end-to-end. That's lower than what experts predicted. Our findings 🧵