AlexM

Scheming might be substantially harder to detect in final model checkpoints than in intermediate checkpoints during training If so, traditional pre-deployment evals will be insufficient to assess risk We need Training-Run Assessments

@harris_edouard Yes. I mention this in the opening paragraph :p I argue later in the post why 3rd parties should also do this

@AlexMeinke Don't the main labs already do versions of this?

Scheming might be substantially harder to detect in final model checkpoints than in intermediate checkpoints during training If so, traditional pre-deployment evals will be insufficient to assess risk We need Training-Run Assessments

IP concerns are the biggest blocker In my blog post I give an overview of how the capacity for different kinds of 3rd party Training-Run Assessments can be gradually built up lesswrong.com/posts/3HvvjffA…

Different artifacts need to be assessed: - Intermediate model checkpoints - The data that shaped the model - The decisions that shaped the training pipeline

We’re hiring for two roles on the monitoring / control team that I’m very excited about: 1. AI security & control engineer: Design threat models for adversarial agents, red-team Watcher and prioritize failure modes Watcher should protect against. 2. Research Scientist (Control): Improve our monitoring stack, e.g. by designing automated red-teaming stack, fine-tuning monitors and playing real-life red-team vs. blue-team games within the research team. We want to make Watcher, the best control product for coding agents. If that sounds interesting, please apply!

Great video explaining our recent research collaboration with @OpenAI It goes into a surprising amount of detail, especially for a popular explainer video, and is very accurate throughout the entire video

@AlexMeinke and I are co-mentoring a MATS stream on the Science of Scheming this Autumn. Apply by June 7th! 🧵

We've published a short summary of our monitoring research agenda: apolloresearch.ai/products/a-sca… 1. Build better evaluation datasets for monitoring 2. Automated red-teaming 3. Adversarial training at large scale We're hiring for applied control researchers: jobs.lever.co/apolloresearch…

Scenario: February 2028 all SOTA models are EBMs, but transformer-based agents invented the architecture Question: Will the “transformers are a dead end”-crowd be posting rage-posts or victory-laps?

@jibayarea Great question In the model I wrote down, it's simply how many different environments there are But I don't know what the most appropriate general measure is (where you'd want to account for how different distributions are from another, rather than naively counting)

@AlexMeinke how can you define "more diverse RL" more mathematically?

I wrote a quick blog post about the "Physics of Reinforcement learning" It's a toy model that I've been finding helpful when thinking about whether models will actively seek reward

Under a bunch of toy assumptions, you can find clean log-log plots predicting that more diverse RL should lead to more reward-seekers The post is a bit mathy, but I've been finding it super useful to think in these terms lesswrong.com/posts/9FH49ZgJ…

More details here: apolloresearch.ai/careers/ If you're interested, come talk to me this weekend!

Research Scientist/Engineer (Science of Scheming): Help build the "Science of Scheming" 1) Figure out completely new evaluation techniques, that have chance of resisting evaluation awareness. 2) Design model organisms, study RL dynamics, build toward scaling laws of scheming.

I'm at EAG this weekend Come talk to me if you're interested in building a science of AI scheming, evaluations, red-teaming and especially if you want to work on it full-time.

Apollo is spinning out from fiscal sponsorship into a PBC. We are increasing our efforts in frontier AI safety research, evals and governance. Additionally, we are spinning up a product team to build AGI safety products starting with AI agent monitoring.

Most under-appreciated finding from our research 2025: naturally emergent and explicit reward seeking with no in-context handholding #eval-awareness-after-rl" target="_blank" rel="nofollow noopener">antischeming.ai/snippets#eval-…

Least surprising update of 2025: models notice alignment evals

My favorite read of 2025 was probably "How training-gamers might function (and win)" blog.redwoodresearch.org/p/how-training… More people should read it