Sam Martin

Going forward we plan to iterate on protocols and settings. In particular we are interested in more fuzzy domains relevant to the alignment relevant tasks we care about. Work done with @dswg97, @jacob_pfau, @a_aristizabalm and Simon Marshall

However, the effect size is sufficiently small that we cannot rule out it being noise. It is possible that with more difficult tasks, or more proposer optimization, that critic optimization becomes useful. We leave this for future work.

Research update from the scalable oversight team at Arcadia Alignment! Debate is a proposed training method for aligning models on tasks we don't have labels for. We want to study this empirically: does it work on today's models? lesswrong.com/posts/hb8pv3zy…

*NEW* AI alignment research team! We're announcing the new alignment team @ArcadiaImpact. A London-based team, working closely with @AISecurityInst to tackle 3 ambitious agendas in AI alignment! 👇 🧵

For these experiments and more, check out our paper arxiv.org/abs/2605.12366 or blogpost lesswrong.com/posts/7vpvNM7v…!

We did preliminary finetuning experiments but were not easily able to elicit improved long context performance in a way that generalized far from the training data.

New Anthropic Fellows research: Classifier Context Rot Anthropic monitors for dangerous actions in agent transcripts that are getting very long. Can monitors handle such long transcripts? 🧵

For these experiments and more, check out our paper arxiv.org/abs/2605.12265 or blogpost lesswrong.com/posts/NxGHQ6DW…!

Benefits of 1-token SFT even generalize to improvements in summarization: a model trained with 1-token SFT identifies dangerous actions taken in transcripts at a higher rate than the starting model!

Anthropic fellows research: we use cross-domain generalization to improve LLM monitors. Language models monitor actions taken by other models to spot dangerous misaligned actions. How do we make them good at this task? 🧵