Elizabeth Barnes

@joel_bkr This doesn't seem obvious to me, cheaper reviews can also incorrectly reject good solutions.

think of graders as having time/$ budget for checking solutions. - it is trivial that having larger budget (maintainer vs algorithmic scoring) leads to weakly lower measured success. - how much lower is empirical question. - there are other possible points on budget curve.

@sayashk @METR_Evals I'd be very curious to see (a) human performance on your reliability metric, and (b) transcript examples for what randomly-selected failures look like for the best models, if you have those to hand.

Hey @METR_Evals—love your work, but we think it's the *metric* that's saturated, not the task suite. For example, despite rapid gains in accuracy, we found limited gains in reliability. We'd love to work together to see if this holds up on the time-horizon task suite.

@_alyxya @METR_Evals It's only showing a subset of the points, see the full graph here: metr.org/time-horizons/

@METR_Evals how is the line created? it doesn't look like it's the best fit line on a log plot, where the earlier data points are generally below the line while the later data points are generally above the line, so I would've expected a steeper slope to better fit the points

We're correcting a mistake in our modeling that inflated recent 50%-time horizons by 10-20% (and reduced 80%-horizons). We inappropriately penalized steepness in task-length→success curve fits. This most affects the oldest and newest models, whose fits are less data-constrained.

our existing uplift study design is broken. devs decline to work without AI (participate in experiment, submit tasks for which they expect AI to be helpful, complete tasks randomized to no AI) leading us to underestimate uplift. x.com/METR_Evals/sta…

Our team is stretched thin at the moment! To continue upper-bounding the autonomy of AI agents, and developing evaluations for monitoring AI systems and their propensity to subvert human control, we need more great engineering and research staff. Please apply below or DM me!

@JerryWeiAI The hope here is not nesc *never* train a model with the dangerous capabilities - but you can use the nerfed model for most cases and have much higher security precautions for the dangerous model (e.g. stricter KYC, lower upload bandwidth, reduced employee access)

@JerryWeiAI Surprising - I’d have guessed you can reduce bioweapon capabilities by ~3yrs of progress or 30x time horizon while only hurting a few % of biomedical usage, by targeting small number of most relevant topics (e.g. specific pathogens, reverse genetics, and maybe cell culturing).

An idea that sometimes comes up for preventing AI misuse is filtering pre-training data so that the AI model simply doesn't know much about some key dangerous topic. At Anthropic, where we care a lot about reducing risk of misuse, we looked into this approach for chemical and biological weapons production, but we didn’t think it was the right fit. Here's why. I'll first acknowledge a potential strength of this approach. If models simply didn't know much about dangerous topics, we wouldn't have to worry about people jailbreaking them or stealing model weights—they just wouldn't be able to help with dangerous topics at all. This is an appealing property that's hard to get with other safety approaches. However, we found that filtering out only very specific information (e.g., information directly related to chemical and biological weapons) had relatively small effects on AI capabilities in these domains. We expect this to become even more of an issue as AIs increasingly use tools to do their own research rather than rely on their learned knowledge (we tried to filter this kind of data as well, but it wasn't enough assurance against misuse). Broader filtering also had mixed results on effectiveness. We could have made more progress here with more research effort, but it likely would have required removing a very broad set of biology and chemistry knowledge from pretraining, making models much less useful for science (it’s not clear to us that the reduced risk from chemical and biological weapons outweigh the benefits of models helping with beneficial life-sciences work). Bottom line—filtering out enough pretraining data to make AI models truly unhelpful at relevant topics in chemistry and biology could have huge costs for their usefulness, and the approach could also be brittle as models' ability to do their own research improves.* Instead, we think that our Constitutional Classifiers approach provides high levels of defense against misuse while being much more adaptable across threat models and easy to update against new jailbreaking attacks. *The cost-benefit tradeoff could look pretty different for other misuse threats or misalignment threats though, so I wouldn't rule out pre-training filtering for things like papers on AI control or areas that have little-to-no dual-use information.

@JerryWeiAI I’d be very interested if you can say anything more about how you developed your classifier, how well you actually succeeded at removing the relevant content from the training data, and how you determined the impact on dangerous capabilities.

@JerryWeiAI I do think you’d need to get an expert to design your classifier, I don’t expect just “ask an LLM whether this is about bioweapons” to work that well.

Hm, I feel dubious without more info. Would want to see some transcripts and how they checked for cheating, and how much they iterated against the benchmark. That line shape smells suspicious to me somehow

I think we also demonstrated value of independent review. E.g. see on page 12 (re training pressure on the CoT that disincentivized revealing harmful info): "There had been internal miscommunications about this topic that became clear only in response to questions from METR"

Props to Anthropic, a great precedent for transparency. Esp sharing sensitive details like compute scaleup calculations, expected performance early in RL. Plus employee interviews and any whistleblowing reports. Led to some useful discoveries + higher confidence in final report

Exactly two years ago, I launched @HalcyonFutures. So far we’ve seeded and launched 16 new orgs and companies, and helped them raise nearly a quarter billion dollars in funding. Flash back to 2022: After eight years in VC, I stepped back to explore questions about exponential technology and the future of humanity. Then ChatGPT launched – we’d entered the exponential AI era. The upside of AI is huge – but so are the risks: misalignment, loss of control, cyber and bio-threats, fraud, adversarial misuse, and more. For humanity to thrive in this era, we’ll need to build a resilient and secure world. And we’ll need an ecosystem of both nonprofit and for-profit solutions led by ambitious, thoughtful leaders from business, policy, academia and media. I found myself asking: 1. How do we get the world’s most talented people working on these civilization-scale challenges? 2. What funding model would allow us to support many different types of projects? With those questions in mind, I launched Halcyon. We’re building something a bit unusual: - @HalcyonFutures, a nonprofit and grant fund that helps leaders and entrepreneurs pivot to ambitious, high-impact work. - @HalcyonVC, a VC firm backing for-profit founders tackling the hardest problems in AI security and global resilience. So far we’ve raised $25m in funding for Halcyon’s nonprofit and VC fund. We’ve incubated or provided zero-to-one capital to 16 nonprofits and companies, and helped them go on to raise more than $200m — from @GoodfireAI's interpretability work to @aiunderwriting's AI risk standards and insurance, @TransluceAI's model behavior research, and @SeismicOrg's public opinion research and media. We're a small team of three (me, Shelby Summerfield and @rossmatican) surrounded by a community of advisors and allies that includes leaders from frontier labs, governments, cybersecurity, philanthropy, startups and VC. Over the coming weeks we’ll share more about Halcyon and what we’re building next. If you're working on something we might be excited about, say hi. Visit our website below in comments ⬇️

To donate to METR, click here: metr.org/donate If you’d like to discuss giving with us first, or receive more information about our work for the purpose of informing a donation, reach out to giving@metr.org

The central constraint to our publishing more and better research, and scaling up our work monitoring the AI industry for catastrophic risk, is growing our team with excellent new researchers and engineers. And our recruiting is, to some degree, constrained by our fundraising.

METR is a non-profit research organization, and we are actively fundraising! We prioritise independence and trustworthiness, which shapes both our research process and our funding options. To date, we have not accepted funding from frontier AI labs.

Today I’m launching @Irregular (formerly Pattern Labs) with my friend and co-founder Omer Nevo: Irregular is the first frontier security lab. Our mission: protect the world in the era of increasingly capable and sophisticated AI systems.

@__nmca__ @DavidSKrueger Yep that is just a graph fuckup, top of CI should be at about 20% if you include only the 15 we thoroughly reviewed (but we skimmed a larger number and thought that probably none of them seemed mergeable)

@BethMayBarnes @DavidSKrueger you could have used a beta so that the CI wasn’t zero width fwiw

Wow. 0% PR success rate. Seasoned AI researchers are familiar with such "generalized Moravec's paradoxes" and so are suspicious of benchmarks that purport to measure progress towards AGI. The safety community has not yet internalized these lessons, but this is progress.