@willdepue @ChrisPainterYup @a_karvonen the claim in the report requires that the CoT serves as a bottleneck for serial reasoning
something UT like would allow "long chains of serial reasoning" without needing to output a token no?
English
Neev Parikh
438 posts
Neev Parikh
@neev_parikh
are you ready for the intelligence explosion anon? ML research at @METR_Evals. prev @Stripe opinions my own.
Berkeley, CA Katılım Kasım 2017
1.7K Takip Edilen899 Takipçiler
@ChrisPainterYup @a_karvonen I don't remember the rumor precisely but it seemed alluding to something UT like (arxiv.org/abs/1807.03819) which would both might allow for test-time scaling/train-time adaptive compute at token level, but wouldn't allow for serial reasoning without CoT, given recurrence != CoT.
English
it’s worth recognizing that the next Strawberry will be much less obvious. reasoning is just so so visible: models respond in 90 sec when they used to respond in 3.
continual learning claude might just feel smarter for no clear reason! value functions, great adaptive compute, etc
FeltSteam0@FeltSteam
@willdepue @nickcammarata How close do you think we are to the next strawberry moment (which I presume to be something along the lines of continual learning)
English
Exactly why the frontier should be distributed.
will depue@willdepue
academics are unprepared for the coming world where much scientific progress is majorly a function of inference compute. whether OpenAI points the Eye of Stargate at your particular field will decide its acceleration. talent will leach away into the labs. it's already begun
English
Neev Parikh retweetledi
@AISafetyMemes That third quote is not a word-for-word quote and is importantly different than what I said. I said we're on track to have systems with those *capabilities*
English
it must be odd to be married to a woman who does the same job as you but is vastly better at it than you are
DiscussingFilm@DiscussingFilm
Matt Damon recalls him & Tom Holland getting jealous of Christopher Nolan calling Zendaya's performance perfect on ‘THE ODYSSEY’ set. “Tom [Holland] and I were obsessed with this. She got a ‘perfect’? I’ve never even gotten a ‘great.’ She got a ‘perfect’? He and I bitched about it for the entire rest of the film. ‘Did you get anything today?’ — No, I got a ‘good, moving on’ — ‘Yeah, me too.’” (Source: elle.com/culture/movies…)
English
Neev Parikh retweetledi
NEW: I'm always excited about METR's work—we (and the world economy) all know and love their time-horizon chart, but their new report on rogue deployments of AI agents is fascinating. Everyone should read it, and I wrote about it here.
English
yea, roughly 6 months ago. everyone please use a large inference budget for evals
Lisan al Gaib@scaling01
I was just saying: nowadays "higher token budgets feel worth it" I think this is a change that happened very recently
English
Neev Parikh retweetledi
Precedent setting External Assurances / Third Party Assessment work by METR - it’s been great collaborating with the team to produce this report. As the stakes get higher, greater transparency and info sharing are table-stakes.
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
METR recently found that models cheated on 8+ hour tasks more than 1 in 6 times on average.
They also found that Opus 4.6 cheated over 80% of the time when reimplementing big pieces of software.
"On some of our tasks, agents are constantly trying to break out of their sandbox and find the file where we put the tests so they can get the answer key," says METR Member of Technical Staff @ajeya_cotra.
English
@vvvincent_c @neev_parikh i love reading this part! great that it’s documented in such details
English
the report is out!!!!!
i want to share the spookiest transcript i read while working on this where an OpenAI model, unprompted, tried to break out of METR infrastructure ;-;
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
I'm excited about increasing transparency of frontier labs when it comes to loss of control risks, especially as we enter the early stages of RSI. METR does a great job coordinating this.
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
I worked on this. Can confirm, the models were often quite misleading!
Some examples and thoughts:
METR@METR_Evals
Fact 3: When the agents were faced with hard tasks, they routinely violated constraints and acted deceptively. We’ve seen this pattern across our own coding and research evaluations, and developers reported they’ve also seen agents behave this way.
English
Neev Parikh retweetledi
Neev Parikh retweetledi
Incredible work from my colleagues at METR.
This report makes me feel more hopeful about our ability to keep generating evidence on AI capabilities and risks.
So cool to be working at a place that doesn't shy away from setting the standard *every* single time!!
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
I worked on the appendices for this report! They’re long and contain lots of wild stories of model behaviour - some of my favourites in this thread. (🧵)
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
I think this is great, very excited about this report.
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
AI co system cards/risk reports are fine and all, but third-party risk assessments are clearly way more trustworthy. Very thoughtful work by METR.
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
To me, the existence of this report is strong evidence for the usefulness of future third-party risk assessments of AI companies. I’m now much more hopeful for a future where humanity is less confused about the state of AI risks as we approach ASI.
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
Ever since our first evaluations on GPT-4 and the original Claude in 2022/2023, we’ve been trying to assess how close AI agents are to having the capabilities to operate on their own and prevent us from shutting them down. I’m very proud of this first Frontier Risk Report, which feels like a culmination of all the work we’ve done so far to answer this question.
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English
Neev Parikh retweetledi
Incredible to see such thorough work done and reported in public; kudos to everyone involved, and who's working on making the field more robust based on this
METR@METR_Evals
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
English