Che-Ping Tsai

Models are typically specialized to new domains by finetuning on small, high-quality datasets. We find that repeating the same dataset 10–50× starting from pretraining leads to substantially better downstream performance, in some cases outperforming larger models. 🧵

A year of cooking 👨‍🍳and we’re finally serving Mamba-3. What began as a small effort to revisit a few recurring limitations of SSMs grew into a much bigger project. Taking a more principled state space perspective ended up tying these threads together.

@dylanjsam @yus167 @OpenAI @zicokolter @andrew_ilyas @furongh Congrats!

I defended my PhD thesis! Also, a very (~4 month) late life update, but I've joined @OpenAI to work on safety research and pretraining safer language models! 📈 Thank you to my advisor @zicokolter and my committee: Matt Fredrikson, @andrew_ilyas, and @furongh! 🙏

I'll admit, going in I was not 100% sure this was possible: we trained a tiny 4B model (QED-Nano) to prove math theorems at the Olympiad level! Today, we release the full recipe, from the data curation done for SFT to our RL algorithm that explicitly optimizes for test-time scaling over millions of tokens (i.e., we train QED-Nano to continually improve as we apply modern day test-time scaffolds like DeepSeekMath-agent over it). 🧵⬇️

RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) 🧵

We run online RL on a mixture of problems: some are easy to explore (high pass rate), and some are very very hard (need to sample A LOT before we see any positive sample). Turns out RL on such a mixture can lead to a "rich-gets-richer" effect, where RL over-sharpens on the easy problems, at the cost of getting stuck in a "plateau" on harder ones, making it even harder to sample a correct trace on those. RL literature calls this "ray interference". In our recent work POPE, we show that using privileged info. to guide exploration on hard problems can tackle ray interference! 🧵⬇️

RL training of LLMs spends tons of compute on sampling rollouts 🤖💸 But most runs are YOLO 🤟, telling us little about how to scale sampling compute optimally. Given a fixed sampling compute budget, how should we allocate it across: • sequential iterations ⏩ • parallel rollouts 🎲 Answers to this with scaling laws 📈 and more in our new blog post ⬇️

1/⚠️ Parallel test-time scaling (e.g., pass@k) usually wastes compute - models often repeat the same dominant failure❌ How should we effectively generate creative solutions? While typical methods such as increasing temperature 🌡️ usually fail, we put forward Mode‑Conditioning (ModC) - a simple yet powerful training and test-time framework that allocates compute across diverse reasoning modes🎨We show that ModC largely improves pass@k across SFT, distillation, and RL settings. With ModC, we get 4-8x efficiency gains in math reasoning using the same training data!

Worked closely with Burak on SSL—super thoughtful, rigorous, responsible and supportive. Strongly recommend them for any academic position!

New SOTA self-supervised learning objective with theoretical guarantees. Worth a look!

🧠🚀 Excited to introduce Supervised Reinforcement Learning—a framework that leverages expert trajectories to teach small LMs how to reason through hard problems without losing their minds. 🤯 Better than SFT && RLVR. Read more: huggingface.co/papers/2510.25… #llms #RL #reasoning

🤖 Robots rarely see the true world's state—they operate on partial, noisy visual observations. How should we design algorithms under this partial observability? Should we decide (end-to-end RL) or distill (from a privileged expert)? We study this trade-off in locomotion. 🧵(1/n)

💡Can we trust synthetic data for statistical inference? We show that synthetic data (e.g. LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moments of synthetic data and those of real data

Consistency models, CTMs, shortcut models, align your flow, mean flow... What's the connection, and how should you learn them in practice? We show they're all different sides of the same coin connected by one central object: the flow map. arxiv.org/abs/2505.18825 🧵(1/n)

Even with full-batch gradients, DL optimizers defy classical optimization theory, as they operate at the *edge of stability.* With @alex_damian_, we introduce "central flows": a theoretical tool to analyze these dynamics that makes accurate quantitative predictions on real NNs.

Excited to share two papers accepted to #NeurIPS2025 ! 1️⃣ Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction We introduce TTI, an RL algorithm that scales the number of interaction steps beyond thinking tokens per step. Our agents learn to act longer➡️richer exploration➡️better success Paper: arxiv.org/abs/2506.07976 2️⃣ Content-Adaptive Tokenizer (CAT) We develop an image tokenizer that adapts token count based on image complexity, offering flexible 8x, 16x, or 32x compression! Importantly, we use just captions (no pixels!) to guide tokenization, enabling adaptive representation for text-to-image generation. Paper: arxiv.org/abs/2501.03120 Look forward to seeing everyone in SD!

🚨Excited to introduce a major development in building safer language models: Safety Pretraining! Instead of post-hoc alignment, we take a step back and embed safety directly into pretraining. 🧵(1/n)

LLMs lose diversity after RL post-training, and this hurts test-time scaling & creativity. Why does this collapse happen, and how can we fix it? Our new work introduces: 🔍 RL as Sampling (analysis) 🗺️ Outcome-based Exploration (intervention) [1/n]

[0/3] 🚀 Introducing Verlog – an open-source RL framework built specifically for training long-horizon, multi-turn LLM agents. 📊 Max episode length comparison: •VeRL / RAGEN → ~10 turns •verl-agent → ~50 turns •Verlog (ours) → 400+ turns 🔥 ⚙️ Technical foundation: •Built on top of the VeRL •Tested on the BALROG benchmark (BabyAI, BabaIsAI, Crafter) •Followed design principles from pytorch-a2c-ppo-acktr-gail 💡 Why Verlog? •For researchers: Skip the heavy engineering. We give you a strong, validated baseline for long-horizon, multi-turn LLM agent across diverse tasks. •For developers: Train on your own long-horizon environments with minimal setup. •Algorithmic edge: With a well-trained value function as an intermediate supervised signal, rollouts can be truncated at any point and still be used for learning. This reduces GPU idle time and boosts training efficiency. This is a genuine advantage of PPO over the GRPO family, widely recognized and leveraged in classic RL, yet often overlooked in LLM agent frameworks. Key features 🧵👇

Self-Questioning Language Models: LLMs that learn to generate their own questions and answers via asymmetric self-play RL. There is no external training data – the only input is a single prompt specifying the topic.