Michael Y. Li

Can a language model learn, end-to-end, what to keep in its own KV cache and what to throw away? Can it learn to forget while it learns to reason? Deep learning's central lesson: capability emerges from end-to-end optimization, not heuristics/strong inductive biases. But for efficiency, we rely heavily on hand-designed approaches. 🗑️ Introducing Neural Garbage Collection (NGC): we train a language model to jointly reason and manage its own KV cache, using reinforcement learning with outcome-based task reward alone. No SFT, no proxy objectives, no summarization in natural language. New paper with @jubayer_hamid, Emily Fox, and @noahdgoodman!

Fixed point iterations for parallelizing nonlinear dynamics is all the rage: - Newton for RNNs - Picard for diffusion models - Jacobi for parallel decode of LLMs But how do these techniques relate, and when should you use them? We show you how in our new paper 🧵

What is the right data mix, and how do we find it as the data keeps changing? This is a core, unsolved problem in continual learning. To tackle it, we built a data mixing algo that works everywhere — pretraining, midtraining, instruction tuning Introducing: On-Policy Mix 🧵1/6

Check out this awesome demo from Omar!

AlphaFold-based models like Boltz-2 and BioEmu train on atomic conformational structures in order to predict protein dynamics. But is it possible to train these models directly on cryo-EM map ensembles, harnessing conformational data that is typically not deposited in the PDB? Introducing CryoSampler: a new approach for fine-tuning Boltz-2 with raw supervision on cryo-EM map ensembles. 1/6🧵

@probablynotaz9 Awesome work, Anthony! Turns out defining the MDP carefully is important!

🚨 Solo-author ICML paper alert 🤫 Ever wanted to post-train your diffusion LLM with good old policy gradients, without having to deal with ELBOs or surrogates? In Simple Policy Gradients for Reasoning with Diffusion Language Models, we show how to make this tractable in a straightforward way. Our framework, Amortized GRPO (AGRPO), lets the model learn from unbiased PG updates via timestep estimation, naturally aligning with dLLM inference while remaining efficient + scalable. Paper: arxiv.org/abs/2510.04019 Code: github.com/probablyabot/a… 1/n

@Infopulsed Thank you! And yes, we're super excited about the future directions.

@michaelyli__ very bullish on this line of work.... it's really incredible

Can a language model learn, end-to-end, what to keep in its own KV cache and what to throw away? Can it learn to forget while it learns to reason? Deep learning's central lesson: capability emerges from end-to-end optimization, not heuristics/strong inductive biases. But for efficiency, we rely heavily on hand-designed approaches. 🗑️ Introducing Neural Garbage Collection (NGC): we train a language model to jointly reason and manage its own KV cache, using reinforcement learning with outcome-based task reward alone. No SFT, no proxy objectives, no summarization in natural language. New paper with @jubayer_hamid, Emily Fox, and @noahdgoodman!

Self-play led to superhuman Go performance, why hasn’t it for LLMs? In practice, long run self-play plateaus like RL. We study why this happens, and build a self-play algorithm that scales better. It solves as many problems with a 7B model as the pass@4 of a model 100x bigger.

I think more is still needed for a good neural memory, but, nevertheless, this is a pretty cool step one!

RL fine-tuning often prematurely collapses LLM entropy. Poly-EPO is a scalable set-RL algorithm that optimizes for a set of accurate solutions with diverse reasoning strategies. Paper: arxiv.org/abs/2604.17654

@FrancoisChauba1 Thanks! And for sure!

@michaelyli__ nice @michaelyli__ ! congrats!! now lets do it w zero order e2e!

@samchenn_ Not eliminate entirely — using them judiciously and remove when appropriate? Also re our discussion earlier, bullish on them more broadly for scientific applications!

@michaelyli__ This guys big on eliminating inductive biases🔥

There's a pretty easy way to relax this design choice. You can introduce some W_e's that map the hidden states to "e"s instead of using the q's to perform scoring. And you can initialize the W_es from the W_qs. More broadly, we think there's a lot to explore in the design space of how to richly parameterize this scoring mechanism.

Using the LM's own attention as the eviction score is the clever piece — you repurpose a signal pre-training already built, instead of bolting on a new head. The question is whether the RL signal eventually warps those scores away from their reasoning-time function, creating a tension between "what to attend to" and "what to keep."

@chrmanning Thanks Chris! And totally agree, we're excited about a bunch of followup directions!

@xavierjgonzalez Spot on summary Xavi, thanks for your interest in our work!

Amazing paper. @michaelyli__ casts clearing the context--which is the most important thing you can do to help your agents to do well for long tasks--as a form of reasoning. This reasoning can be learned via RL. No messy heuristics. Just the magic of end-to-end learning

@singhh5050 Thanks Harsh!

@michaelyli__ The coolest work ever!!

@fabianfranz Thank you for your interest in our work!

@michaelyli__ Thanks for giving AI agency and working WITH the models. Very nice!

@ericzelikman Thanks Eric! ❤️

@michaelyli__ cool work!

@JayaGup10 COLLECTION!!!!

@michaelyli__ GARBAGE!!!!

@AdityaCowsik Indeed.

@michaelyli__ This really shows how much engineering you need to fully accept the bitter lesson!