Nate Rahn

@viemccoy @allylyq @JonathanMi98298 @sleight_henry @ErikJones313 @EthanJPerez Thanks! Blog post is available at: alignment.anthropic.com/2026/abstracti… and full paper at: arxiv.org/abs/2602.12318

@n8rahn @allylyq @JonathanMi98298 @sleight_henry @ErikJones313 @EthanJPerez Very interesting, will there be a full writeup anywhere?

Current pre-deployment evals face a trade-off. Static evaluations based on fixed prompt sets are too weak, missing rare failures across the vast space of possible user queries. Adversarial prompt optimization is strong, but narrow: it finds very specific prompts that are unlikely to appear in the wild. Categories bridge this gap.

This project would not have been possible without the great work of my co-lead @allylyq, collaborators Avery Griffin, @JonathanMi98298, @sleight_henry, and excellent research supervision from @ErikJones313. Finally, we thank @EthanJPerez for spearheading the Anthropic Fellows Program. I could not have asked for a better environment to do this research. Grateful to all involved!

We believe our results are an important step toward realistic pre-deployment auditing of model character. Beyond the simple identification of character failures, we are optimistic that the interpretability of categories could help model developers iterate on constitutions, generate safety training data, and anticipate deployment risks, all before a single real user interacts with the model. Read on to learn more… Blog post: alignment.anthropic.com/2026/abstracti… Full paper: arxiv.org/abs/2602.12318

New Anthropic Fellows research: Abstractive red-teaming of language model character The worst way to find out about a character flaw in your language model is from a viral screenshot. How can we find these issues before deployment, rather than after? In this work, we introduce abstractive red-teaming, a new approach that searches over natural-language categories of queries, rather than individual prompts.

We’re hiring someone to run the Anthropic Fellows Program! Our research collaborations have led to some of our best safety research and hires. We’re looking for an exceptional ops generalist, TPM, or research/eng manager to help us significantly scale and improve our collabs 🧵

Late update: I’ve moved to the Bay Area for a 6-month research fellowship at @AnthropicAI ! I’d be glad to meet other researchers working on RL for language models, agents, subtle and unverifiable rewards, etc. — DMs open.

@cong_ml @GoogleDeepMind Congrats Cong! Good luck!

Extremely happy to share that I've joined @GoogleDeepMind as a Research Scientist on the Open-Endedness Team! Looking forward to seeing old friends again and making new ones, do let me know if you are in London! 🫶

Proud to have been part of the team behind Meta Motivo, a truly groundbreaking foundation model for behavior. It’s the first of its kind, enabling you to instantly generate human-like behaviors for any reward function or goal. Make sure to check out the demo for yourself!

@Xidong_Feng Yeah, I've wondered the same thing. I suspect it reflects the academic background of the authors / community being targeted.

A question about the Process reward model in a lot of LLM reasoning papers: Why almost no paper call automatic PRM dataset building (e.g., the pipeline in Math-shepherd) Monte-Carlo value function estimate (term from RL)? They are exactly the same thing.

@Allen_A_N Hey Allen, nice paper! It's cool that you can tune LLMs to be near-optimal. In case you haven't seen it, you might also be interested in our recent work which considers LLM exploration through the lens of representation-level steering: arxiv.org/abs/2406.00244

LLMs are in-context RL learners, but not great because they can’t explore well. How do we teach LLMs to explore better? 🤔 🔮 Solution: Supervised fine-tuning on full exploration trajectories. Preprint with GDM: arxiv.org/abs/2410.06238 🧵

@giomonea Hey Giovanni! Nice work, I like the study of modulating exploration through intervening on the context. You might also be interested in our recent work which studies LLM exploration through representation-level steering: arxiv.org/abs/2406.00244

ICL has proved phenomenal at improving LLMs, but requires access to gold labels (supervised learning). In our new preprint, we find that LLMs can also learn in-context via predictions and reward signals only (via reinforcement learning)! 🧵 📝 ArXiv: arxiv.org/abs/2410.05362

With today's announcement @karlmoritz, Richard & I are thrilled to launch Reliant's next phase - building AI that will completely change how we work with data. Excited to bring Tola Capital, @inovia, and @mavolpi's expertise & experience on this journey. PS: We're hiring :)

New #RLC2024 paper Three Dogmas of Reinforcement Learning joint w/ @mark_ho_ and @aharutyu! arxiv.org/pdf/2407.10583 We reflect on where our scientific paradigm needs adjustment, and suggest three departures from previous conventions. Curious to hear what folks think! 🧵

Off to #ICML2024 to present our work on “Controlling Large Language Model Agents with Entropic Activation Steering” at the mech interp wkshp. Would love to meet folks curious about understanding/improving LLM agents, steering vectors, etc - DM or email me if you'd like to chat!

Did you miss the recent Auroras? No problem! ✨🎆 Super excited to share AURORA, a *general* image editing model + high-quality data that improves where prev work fails the most: Performing *action or movement* edits, i.e. a kind of world model setup Insights/Details ⬇️

Excited to share the latest paper of my PhD, looking at steering LLM agents. Check out the thread!