Rishi Mehta
281 posts
Rishi Mehta
@rishicomplex
Solve i̶n̶t̶e̶l̶l̶i̶g̶e̶n̶c̶e̶ ̶ coding, use it to solve everything else | Research @AnthropicAI | Past: RL @GoogleDeepmind: AlphaProof co-lead, Gemini.
San Francisco, CA Katılım Temmuz 2009
335 Takip Edilen3.5K Takipçiler
Personal update: I've joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.
English
Sara's team is awesome, apply if you're excited about aligning Claude!
Sara Price@sprice354_
This is important and challenging work. If you are excited about contributing please consider applying - particularly by joining the Anthropic Fellows program!
English
what's up with the X algorithm it's just cycling the same 10 posts for me
English
Rishi Mehta retweetledi
at long last we have built and chosen not to release the zero-day machine from the classic sci-fi tale “please do not release the zero-day machine”
English
Claude Mythos Preview is a substantial jump, especially on cyber, and requires a new kind of cooperation for society to adapt
Anthropic@AnthropicAI
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software. It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans. anthropic.com/glasswing
English
@sideboared @FakePsyho In the case of humans, per the quote in the paper it appears they can reset the action count
English
@rishicomplex @FakePsyho You can read the models' play logs on the site. They can and do hit reset when they feel they need to. Don't know if it affects their score different from any other move though.
English
AI (or any human) will never get 100% in ARC-AGI-3
Let me introduce you to the worst game mechanic you can find in a puzzle game: fog of war
At the start, if you go right instead of bottom, you're wasting many moves. Your score on this level literally depends on a conflip!
English
@andreasorob @fchollet In the case of the human participants, from the quote in the paper it appears they can reset the action count in the middle of a game, which the AI can't do
English
@rishicomplex @fchollet Yes, the AI is also allowed to reset the level (neither can reset the game): "Competition Mode... Only Level Resets are premitted..." #competition-mode" target="_blank" rel="nofollow noopener">github.com/arcprize/arc-a…
English
@fchollet according to your paper: "Participants were limited to a single attempt per environment and could not revisit previously completed levels. However, they were allowed to reset the current level at any time. In some cases, participants reset levels after reaching a solution in order to improve efficiency, though this typically increased total interaction time."
So humans could play around with the task a bunch, and then just reset the game when they figured it out to get the optimal trajectory? Is AI allowed to do this?
François Chollet@fchollet
ARC-AGI-3 is out now! We've designed the benchmark to evaluate agentic intelligence via interactive reasoning environments. Beating ARC-AGI-3 will be achieved when an AI system matches or exceeds human-level action efficiency on all environments, upon seeing them for the first time. We've done extensive human testing that shows 100% of these environments are solvable by humans, upon first contact, with no prior training and no instructions. Meanwhile, all frontier AI reasoning models do under 1% at this time.
English
@RyanPGreenblatt Possibly not because it looks like they cheated by giving humans infinite retries x.com/i/status/20373…
Rishi Mehta@rishicomplex
@fchollet according to your paper: "Participants were limited to a single attempt per environment and could not revisit previously completed levels. However, they were allowed to reset the current level at any time. In some cases, participants reset levels after reaching a solution in order to improve efficiency, though this typically increased total interaction time." So humans could play around with the task a bunch, and then just reset the game when they figured it out to get the optimal trajectory? Is AI allowed to do this?
English
I wish they published the performance for each human baseliner rather than just the performance of the second best human run on each task.
My current guess is that the median human baseliner would score around ~15% on the metric but we can't check because the data isn't public!
ARC Prize@arcprize
Announcing ARC-AGI-3 The only unsaturated agentic intelligence benchmark in the world Humans score 100%, AI <1% This human-AI gap demonstrates we do not yet have AGI Most benchmarks test what models already know, ARC-AGI-3 tests how they learn
English
Why not build data centers in the sea before sending them to space? It seems strictly easier, and cooling wouldn’t be the bottleneck. What am I missing?
Sawyer Merritt@SawyerMerritt
NEWS: Nvidia CEO Jensen Huang announced today that the company is working on a new chip/computer for orbital data-centers called Nvidia Vera Rubin Space-1. "It's going to start data-centers out in space. Of course, in space there's no conduction, no convection, there's just radiation, so we have to figure out how to cool these systems out in space, but we got lots of great engineers working on it."
English
I've decided to leave OpenAI. I'm incredibly proud of all the work I've been part of here, from helping create the reasoning paradigm with @MillionInt, scaling up test-time compute with @polynoamial, working on RL algorithms with my fellow strawberries, shipping o1-preview (which started life as of one of my derisking runs), to post-training o1 and o3 with @ericmitchellai, @yanndubs and many others. I'm most proud of having led the post-training team here for the last year -- the team has done incredible work and shipped some really smart models, including GPT-5, 5.1, 5.2, and 5.3-Codex. OpenAI has genuinely some of the most talented researchers I have ever met, and I have learned more than I could have imagined knowing since I joined as a new grad.
I want to thank @markchen90 @FidjiSimo @sama @merettm for all their support over my time here, and too many collaborators to name for the insights, ideas, and just plain fun we have had working together. After leading post-training for a year, though, I'm longing to start fresh and return to IC research work. I've been thinking about going back to technical research for quite some time, and I genuinely believe my colleagues and team here are set up to succeed going forward without me.
I'm personally very excited for my next chapter -- I'm proud to be joining @AnthropicAI to get back into the weeds in RL research, and I'm looking forward supporting my friends there at this important time. Many of people I most trust and respect have joined Anthropic over the last couple of years, and I'm excited to work with them again. I have also been very impressed with Anthropic's talent, research taste and values, and I'm excited to be part of what the company does next!
English
@delam25 This post explains some nuances on autonomous weapons, not using Claude for this right now doesn't give China an edge anthropic.com/news/statement…
English
@rishicomplex Ironic that your last post expressed concern about China gaining an edge
English
Worth reading the linked article about distillation by Chinese models - this is the biggest threat to the US AI lead
Anthropic@AnthropicAI
We’ve identified industrial-scale distillation attacks on our models by DeepSeek, Moonshot AI, and MiniMax. These labs created over 24,000 fraudulent accounts and generated over 16 million exchanges with Claude, extracting its capabilities to train and improve their own models.
English