Micah Carroll

Apollo folks are incredibly sharp and hard working and it’s been a joy and honor to collaborate with them this past year and a half. If you are looking for an impactful role in AI safety it would be hard to do better IMO!

Chain of thought monitors are a key layer of defense against AI agent misalignment. To preserve monitorability, we avoid penalizing misaligned reasoning during RL. We found a limited amount of accidental CoT grading which affected released models, and are sharing our analysis. alignment.openai.com/accidental-cot…

Bad that it happened, but really glad they're noting this kind of mistake publicly - we need enough companies doing it systematically that we can assess , understand and improve industry practices and performance, as well as get a realistic sense of how low the mistake rate can get under high competitive pressures. Reminder that we should reward the behaviours we want to see more of and punish the behaviours we want less of. If we make it too costly for companies to admit mistakes, we get less information, not fewer mistakes.

@NurmiWilliam Yes

@MicahCarroll Wait, you reran training for a (largely?) non-thinking model to confirm that chain of thought leak does not affect monitorability?

We recently found some instances of CoT grading during the training of previously deployed models after building a system that scans all OpenAI RL runs for accidental CoT grading. We did not find clear evidence that these instances degraded CoT monitorability.

We reviewed OpenAI's blog post “Investigating the consequences of accidentally grading CoT during RL". blog.redwoodresearch.org

Glad more AI safety work is getting reviewed by independent parties. Most lab posts never go through peer review, and when they do it's with a 4+ month lag: an eternity in AI terms. Public reviews like Buck'scan help surface methodological gaps and keep researchers honest.

@redwood_ai @apolloaievals @METR_Evals @bshlgrs @RyanPGreenblatt @BronsonSchoen @MariusHobbhahn @neev_parikh And @redwood_ai's blogpost by @bshlgrs on this here: blog.redwoodresearch.org/p/openai-cot

@redwood_ai @apolloaievals @METR_Evals @bshlgrs @RyanPGreenblatt @BronsonSchoen @MariusHobbhahn @neev_parikh Find the full blogpost here: alignment.openai.com/accidental-cot…

Bad news: We at OpenAI have recently found we were accidentally putting optimization pressure on chain of thoughts Good news: It didn’t affect monitorability and it seems that degrading monitorability via CoT grading is harder than we thought