VulcanBench

When a new model comes out, I really want to test the heck out of it so I can determine if/how it fits into the agentic coding workflow for my eng team. Since Sonnet 5 came out yesterday, I wanted to jump on benchmarking right away. I used @VulcanBench to run evals on Claude Sonnet 5 vs Opus 4.8 on 4 real, decontaminated OSS bug-fixes overnight. Both solved all 4, which I expected, but the cost was not at all what I expected. Over the last few months I've found that for routine coding tasks, wayyyy too many people use Opus when Sonnet can handle it just fine. But I got a result I wasn't expecting, one of the tests was on a larger repo, 76,000 lines of code, and Sonnet 5 burned $15.15 vs Opus's $2.08 (7x). The nugget I took away from this is: Sonnet thrashes exploring big codebases, Opus doesn't seem to have this problem. More details analysis below.

Just finished my first set of benchmarks comparing Sonnet 5 and Opus 4.8 on @VulcanBench across effort levels. This is the first of a number of benchmarks I plan on doing with Sonnet 5, but I wanted to get something kinda unique and different out there. And yes, don't worry, I'll do a GLM 5.2 comparison. I don't think many benchmarks are looking at how changing effort levels impacts accuracy, so I thought this could be an interesting angle to start with. In total I ran 936 test runs across both models, and three different effort levels. Here's the high-level results, will be sharing a more detailed overview in a full report tomorrow: 💸 Sonnet 5 matches Opus 4.8's accuracy at roughly HALF the cost per run. 🎯 Sonnet 5 at high effort is the only config to solve all 52 tasks (100%), and it still costs less than Opus at high effort. 📈 Reasoning effort scales Sonnet 5 (97 to 100%) but does nothing for Opus 4.8 (flat 97 to 98%). Extra thinking is wasted on Opus here. 🧮 Every single Opus cell is Pareto-dominated on cost vs quality. ⚡ Opus's one edge: fewer tokens, faster runs. Sonnet "thinks" ~2x harder at high effort but wins on price anyway (its tokens are 3/5 the cost). ⚠️ Honesty check: this suite saturates the frontier, so the accuracy gaps are tiny (1 to 3 tasks). The real signal is cost and how each model responds to effort, not raw capability. And please remember, I'm not some kind of benchmarking expert, this is new territory for me so I'm testing and learning as I go. Still working on updated evals that will be harder for these models so I can start to get scores in the 80% - 90% range. That being said, some interesting insights from this benchmark run, and gives me a lot more ideas for the next run!