Vinay Samuel

Post-training makes LLMs safer and better at following instructions, but less diverse. 🤔 Can we get that diversity back without sacrificing alignment? Introducing ReDiPO: a preference optimization recipe for restoring distributional diversity while preserving safety and instruction-following.

Can open-weight language models write long stories? Not well: either their writing quality degrades at longer lengths or they severely undershoot in length as the requested length grows, if not both. Frontier models fare better. This begs the question, can we train open-weight language models to write long stories with limited data and compute?

Paper: arxiv.org/abs/2605.30021 Code/data: github.com/vsamuel2003/Re… Work with @YapeiChang and @MohitIyyer

Takeaway: alignment should not mean model collapse for distributional diversity. Base models can be reservoirs of valid alternatives that instruction tuning pushes down. ReDiPO shows how careful preference data can bring those modes back into instruct models. 🌱

Across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, ReDiPO boosts NoveltyBench distinct_k over instruct checkpoints by: 📈 +134% 📈 +33% 📈 +44% These novelty gains do not compromise instruction following or safety alignment on all tested models.🛡️

Post-training usually optimizes one answer at a time: helpful, safe, well-formatted. But for brainstorming, creative writing, and hypothesis generation, users often need many distinct valid answers, not multiple samples of the same idea. ReDiPO uses the base model as a source of missing alternatives. For each prompt, it samples base + instruct responses, rewrites base outputs in instruct style, filters unsafe/low-quality candidates, then forms DPO pairs where quality is matched but marginal diversity differs. 🔁

Post-training makes LLMs safer and better at following instructions, but less diverse. 🤔 Can we get that diversity back without sacrificing alignment? Introducing ReDiPO: a preference optimization recipe for restoring distributional diversity while preserving safety and instruction-following.

Ablations show that marginal-diversity pair selection drives the diversity recovery, base-response rewriting makes candidates usable, and quality/safety filters help preserve alignment. 🧪

Compared with DDPO, a DPO baseline for creative writing, RiDiPO gives up some raw novelty but keeps the model much more aligned: ✅ MTBench: 5.29 vs 7.38 ✅ IFEval: 0.566 vs 0.817 ✅ HarmBench ASR: 0.296 vs 0.075 ✅ Arena-Hard: 0.198 vs 0.287

Excited to announce my first preprint in LM interpretability! Latent reasoning models are not monitorable by default, since they don't reason in human-readable, natural language text. But can we make progress in understanding their intermediate reasoning steps using mech interp?

LM agents today primarily aim to automate tasks. Can we turn them into collaborative teammates? Introducing Collaborative Gym (Co-Gym), a framework for enabling & evaluating human-agent collaboration! I now get used to agents proactively seeking confirmation or my deep thinking.

@omarsar0 Thank you for sharing our work @omarsar0 !

Evaluating Persona Agents and LLMs It's great to see research around evaluating persona agents. Persona agents is probably going to become one of the most common and useful ways to use LLMs but there is very little research on how to properly assess them. This work proposes a benchmark to evaluate persona agent capabilities in LLMs. Finds that Claude 3.5 Sonnet only has a 2.97% relative improvement in PersonaScore compared to GPT 3.5 despite being a much more advanced model. Custom agents that act as specific personas makes sense in domains like education, healthcare, creativity, productivity, entertainment, and more. The interesting thing is that current models are explicitly trained to operate this way but a lot of people are interested in using them like that.

Our evaluation benchmark, details, and code are available on our website at: personagym.com This work was done in collaboration with: @zou_henry43378 , @DataFleuret , @shrechaudhari , @ashwinkalyan7 , Tanmay Rajpurohit, @AmeetDeshpande, @karthik_r_n, @VishvakM

We evaluate 6 LLMs on 200 personas and 10K questions, noting: - Significant potential for improvement (e.g., Claude 3.5 Sonnet's 2.97% PersonaScore gain over GPT-3.5). - Larger, more complex models don't necessarily enhance persona agent capabilities.

PersonaGym – the first platform to evaluate persona agents. Persona agents are critical to magical AI experiences, but how do we evaluate them? PersonaGym uses dynamic LLM-powered environments for faithful and targeted agent eval. personagym.com