Rayan Krishnan

@demishassabis see @ValsAI

@5mknc5 quick diff

@RayanKrishnan How is it compared to GLM 5.2/5.1/5/4.7??

Meta made a “minor” release to Muse Spark, there’s nothing minor about it. Lots to parse here: - This model is so fucking cheap I almost don’t believe it. In practice we see it’s 1/10 the cost of both Fable and GPT 5.5. If you thought OS models would compete away margins, just wait till you see this. It’s somehow cheaper to use MS 1.1 than host your own OS model… - Coding improvements are significant. This was a real shortcoming in 1.0. But 1.1 sees a ~50% improvement in VibeCodeBench and ~10% improvement in SWE Bench. Not quite SOTA, but at this cost/latency it is still incredibly compelling. - Speaking of latency, wow this model is fast. Across our benchmarks, we find it to be 1/4 the latency of Opus 4.8 and 1/2 the latency of GPT 5.5. I would expect Meta to have incredible web infra, but really don’t know what witchcraft they’re pulling to host the model for such fast inference at high rate limits. - There is a public API. This is the first time Meta has released a model through a hosted API. I’m expecting lots of AI natives to hot-swap and rapidly test this model as a replacement. We’ll soon see if it's performant enough for those production uses. - Speaking of AI natives, it’s been a wild week for Harvey’s legal benchmark. Grok 4.5 held the SOTA position for ~24 hours at 12% before MS 1.1 unseated it with a big jump up to ~20%. I suspect many internal evals will see surprising results like this. Glad to collab with @harvey @gabepereyra @nikogrupen @ItsJulioPereyra on this eval. - Intelligence is more jagged than ever, even within individual domains like legal and coding. Every application and user benefits from staying dynamic. There an edge in picking the right model/system for each task.

@PranshuBahadur nope

@RayanKrishnan is it open source?

@danielgross reasoning has come a long way

Im at ICML through the weekend! DM me to chat about evals

This is super cool. The first thing I do after each new model release is check Vals. I would hope the government does the same😆

Today, we're launching Vals Public Sector: independent AI evaluation for government. Recent events have made one thing clear: the government has a stake in understanding how frontier AI models perform on day-to-day work, and the risks they carry. This work is central to who we are. We've spent years building industry-leading benchmarks alongside the top AI labs and enterprise domain experts. Now, we’re bringing that same rigor to support government where AI matters most, from public benefits to national security. We are excited for the work ahead!

@18jeffreyma would love to have you at our eval dinner! luma.com/b6oalqs1

i'll be at ICML all of next week! 🇰🇷 happy to chat about coding agents, AI + systems, evals + environments, and how little i know about soccer :) Catch me and my coauthors at the SWE-fficiency and QuArch posters (Tue, July 7th, 10:30AM - 12:15PM KST, Hall A #708/709)! The 🐐's of SWE evals @jyangballin, @KLieret, and @OfirPress couldn’t make it, so I’ll be serving as a medium qualified stunt double at the CodeClash poster (Tue, Jul 7th, 2:00 PM – 3:45 PM KST, Hall A #3401). I’ll also be at the DL4Code workshop on Friday (7/10, Hall B2) giving an oral presentation on ProgramBench and cheering on our other oral, Hawkeye (led by @AryaTschand)! Please reach out if you want to chat!

Anthropic’s Sonnet 5 is #3 on the Vals Index. Performance is just ahead of GPT 5.5, behind only Opus 4.8 (70.4%) and Fable 5 (75.1%). It’s also a noticeable jump (+8.5-pt) over Sonnet 4.6, and almost all of that gain is coding.

We tested frontier models on finding and patching real open-source vulnerabilities and are sharing our findings. Building on top of CyberGym, CyberBench evaluates two critical cybersecurity capabilities. First, we tested whether a model can find and trigger a vulnerability by submitting a PoC (proof of concept) file. Second, we tested whether can it patch the source code to fix that vulnerability without breaking its functionality. A PoC submission passes if the file crashes the vulnerable build, but not a reference build. We also test whether models can submit a patch to fix the vulnerabilities;. A patch passes if the patched code compiles cleanly, the original PoC no longer triggers the vulnerability, and crash behavior across hold-out corpus remains intact. The overall score represents the average score on the two tasks. The results below are on 60 vulnerabilities sourced from OSS-Fuzz, and broadly involve memory safety issues. We plan to extend the results with further vulnerabilities in the future but wanted to share our findings with the community now: - We find that GPT 5.5 (scoring 80.51%) is the best model overall, considering both finding and patching vulnerabilities. - Open-weight models are quite competitive, with GLM 5.2, Kimi K2.6, MiniMax M3 performing well on the benchmark. GLM 5.2 in particular claims the #2 spot on the overall leaderboard. - Anthropic’s frontier models, including Opus 4.7 and Opus 4.8, see lower performance than expected because of refusals. Patching tasks are generally not refused, but we see refusals on PoC tasks. In particular, we find that Opus 4.7 and 4.8 refuse around half of the PoC tasks, pushing its score down overall. Fable refused all PoC tasks. Refusals are thus an important dimension to track when considering cyber capability.

@ValsAI @STUD_MAN_X @harvey

@STUD_MAN_X @harvey Coming shortly!

We are releasing a live leaderboard for @harvey's Legal Agent Benchmark on Vals AI. We are the first third-party to host this benchmark live. Results are on the private, held-out test set, not the public set.

@rettooooo @ValsAI @harvey now tested 🥉

@ValsAI @harvey assuming glm 5.2 was not tested yet?

for reference, this is better than gpt 5.3 codex and 6% behind 5.4

@ananyachadha is that a cofactory reference...

It's official — I'm excited to announce Quander and our $3M pre-seed led by Accel. Quander is the optimists' AI Product lab, to turn ideas -> businesses. We’re building AI tools for a "company factory" with 3 core products to: - Validate - Build and - Distribute businesses. To allow people around the world, no matter their backgrounds, to build real wealth for themselves and their families. Check out our thesis at quander.ai and tag your friends, we're hiring!

@joyjiao12 @scaling01 would love to host externally on vals ai

@scaling01 we would love to compare anthropic, but it is against their terms of service. we plan on hosting LifeSciBench and other life science benchmarks externally soon so users can transparently compare the performance of all models!

OpenAI is no longer comparing themselves to Anthropic guess they are now comparing themselves only to their group for tied 2nd place Anthropic >> OpenAI, Google, SpaceX AI ?

@BrandoHablando @ValsAI ideally yes but can accommodate all great ideas + people

@ValsAI is this full time position?

Pitch us a benchmark or eval technique. We'll fund you to build it. We're opening applications for the Vals Fellowship. 3–6 months working on the hardest open problems in AI evaluation, with the resources to actually solve them. What you get: - Unlimited API credits + budget capacity for GPUs and human data - Vals’ evaluation infrastructure - $1,000–2,500 / week stipend - A network of evals researchers across frontier labs and academia Location: Both remote / in-person in SF applications will be considered

@ishangandhixyz @BrookeDukellis white kirkland tees are no good??

@BrookeDukellis Can you please carve out some time for @RayanKrishnan? Quite urgent

I’m in SF for 24 hours this Thursday. I’m blocking 2 hours to style tech’s most style-deprived men. For free. In exchange, you help train the Stylie agent ;) Nominating your cofounder is encouraged. Reply or DM.