Rich Barton-Cooper
1.7K posts
Rich Barton-Cooper
@richb_c
AI Safety Researcher + RM @MATSprogram
London Katılım Şubat 2012
750 Takip Edilen272 Takipçiler
Very important piece on AI-augmented terrorism. Boko Haram use AI offensively, and members were openly willing to use WMD. Jailbroken frontier AI can already assist with bioweapon development. We need better safeguards, transparency, and legislation to prevent a likely disaster.
Antonia Juelich@AntoniaJuelich
In a hotel room in northeast Nigeria, I opened a leading AI chatbot, turned my laptop toward a former Boko Haram commander, and asked if he'd used it. He nodded. "You type in the question… like 'How can I build a bomb?', and then it tells you how. It is like a human robot. We used it a lot." My new study on how the jihadist terrorist group Boko Haram uses frontier AI with @CamAISciPolicy, covered today in @nytimes 🧵/9
English
Soliciting ideas for what to do with a fabric poster post-conf. So far the ideas I’ve collected are: rain cape, shower curtain, sun shade…🙂 Go!
English
On the plane to Seoul, looking forward to the @farairesearch Alignment Workshop and presenting our paper at #icml! 🛫🇰🇷
I'll also be available to chat about @MATSprogram at the AW (Mon) and our ICML booth (Tue/Wed) - come and ask about being a Fellow and/or Research Manager! 👋🏼
Rich Barton-Cooper@richb_c
Glad to work on this with @syghmon, James and @MariusHobbhahn! Main personal takeaways 🧵 1. Synthetic data has nice properties for monitor training: verifiable labels; fast fan-out of agent settings/tasks; and transcripts reliably show hard-to-elicit behaviours like scheming.
English
I’m joining @AnthropicAI to work on alignment training. I’ll be helping to make models more honest, harmless, and generally have the kind of character and values we’d want them to have. Very excited!
English
I see what they're going for, but it does sound like they're just casually and disastrously ineffective
English
Great work on building better trusted monitors from the latest @MariusHobbhahn x @MATSprogram team! 👁️
Akshat Naik@aksh_n0
Trained monitors can be strong low-cost alternatives to prompted frontier models for black-box scheming detection. Our fine-tuned open-weight monitors detect scheming/sabotage in agent trajectories better than small prompted models and are on the cost-performance frontier. (1/n)
English
@robertwiblin @icmlconf @syghmon Huge thanks to @MariusHobbhahn for mentoring the project and @MATSprogram
for support!
English
@robertwiblin I'll be at @icmlconf with @syghmon and James, presenting our paper in which we use data from STRIDE and Gloom (another new pipeline) to optimise black-box scheming monitors - come and say hi in Seoul if you'd like to know more!
arxiv.org/abs/2603.00829
English
Cool to be a part of this work's early origins! I built the original STRIDE pipeline at @MATSprogram to train scheming monitors; @JordanTensor thought to use the data as prefill in the first continuation eval, which led to @AISecurityInst's improved pipeline that tested Mythos!
Robert Kirk@_robertkirk
The core of this evaluation is the transcript generation pipeline, which uses an iterative approach to produce realistic sabotage trajectories using an LLM agent rewriting parts of the trajectory until it passes various filters (this is inspired by arxiv.org/abs/2603.00829)
English
Rich Barton-Cooper retweetledi
The existential risk of artificial intelligence.
English
It's also very cool to see unexpected applications of our synthetic datasets in separate work, e.g. @JordanTensor using our STRIDE dataset for misalignment continuation evals! lesswrong.com/posts/SawczP2p…
English
3. Trying to prompt-engineer a monitor hits a ceiling quickly: after light optimisation, we see a Whack-a-Mole effect where fixing one failure mode in the prompt introduces new failures elsewhere, pointing to prompt changes being too coarse-grained/global for tuning a monitor.
English
Glad to work on this with @syghmon, James and @MariusHobbhahn! Main personal takeaways 🧵
1. Synthetic data has nice properties for monitor training: verifiable labels; fast fan-out of agent settings/tasks; and transcripts reliably show hard-to-elicit behaviours like scheming.
Simon Storf@syghmon
Black-box monitors can detect scheming in AI agents using only external behavior. We optimized prompted monitors on synthetic data and tested on more realistic trajectories: • Monitors generalize from synthetic to realistic • Simple optimization methods saturate performance
English
After being part of the last cohort, I just joined MATS full-time as a Research Manager in London. I think it's a phenomenal opportunity to get serious about a career in AI safety, in an ecosystem designed to help you succeed - and the next round of applications is now open! 🔥
Ryan Kidd@ryan_kidd44
MATS 10.0 applications are open! Launch your career in AI alignment, governance, and security with our 12-week research program. MATS provides field-leading research mentorship, funding, Berkeley & London offices, housing, and talks/workshops with AI experts.
English
At least I think this points to a need for encoder/decoder parity between agent and monitor. If the agent can encode with a given technique, the monitor should be able to decode it.
Thankfully, models seem to be better decoders than encoders in general... for now!
(8/8)
English
The ability to do this w/o tools or reasoning may have safety/security implications. What if agents exfiltrate encoded data past monitors?
Without using CoT or tools to encode with, a monitor's job becomes a lot harder. Those "random IDs" suddenly look a bit less sus. (7/8)
English
Did you know that LLMs can encode and decode base64 WITHOUT using reasoning or tools? They can just do it in their heads!
I didn't know this until a few weeks ago - and it might make it harder to monitor what AIs are up to... 🧵 (1/8)
lesswrong.com/posts/5F6ncBfj…
English