Henry Kiss Ehrenberg

3

21

1.4K

Henry Kiss Ehrenberg@henryehrenberg·5d

Distinct trends emerging out of the major labs on Senior SWE-bench. Claude models reach higher overall scores, but at higher token usage. GPT-5.x models are getting better AND more efficient. Would be interesting to see what GPT-5.6 could do at double-secret-max effort.

English

2

11

34

1.5K

Henry Kiss Ehrenberg@henryehrenberg·5d

Looks like a lot more effort going into training for work in modern, multi-service production stacks (vs. OSS libraries). On Senior SWE-bench tasks that touch both Python BE service + Typescript FE code, GPT-5.6 Sol ups the ante again @ 39.1%.

Grok 4.5 and GPT-5.6 Sol join the Pareto frontier on Senior SWE-bench, and there's a clear trend towards efficiency. GPT-5.6 Sol: Opus 4.8 perf @ 40% of the cost Grok 4.5: GPT-5.5 perf @ 25% of the cost Grok 4.5 climbed to #2 on senior-level bug solving but for just ~$1 / task

English

4

26

3K

Henry Kiss Ehrenberg@henryehrenberg·5d

Quick additional note: there was a scoring issue on one Fable 5 trial when we initially reported its results. Its corrected overall score is 29.1%.

English

1

8

289

Henry Kiss Ehrenberg@henryehrenberg·5d

More detailed analysis on senior-swe-bench.snorkel.ai coming soon. Thanks to @OpenAI and @SpaceXAI for early access and the collaboration around testing. And congrats on the launches!

English

10

333

Henry Kiss Ehrenberg@henryehrenberg·5d

Grok 4.5 and GPT-5.6 Sol join the Pareto frontier on Senior SWE-bench, and there's a clear trend towards efficiency. GPT-5.6 Sol: Opus 4.8 perf @ 40% of the cost Grok 4.5: GPT-5.5 perf @ 25% of the cost Grok 4.5 climbed to #2 on senior-level bug solving but for just ~$1 / task

English

6

13

45

9.7K

Henry Kiss Ehrenberg retweetledi

Elon Musk@elonmusk·6d

Grok 4.5 true usefulness is excellent

Snorkel AI@SnorkelAI

We obtained early access to evaluate @SpaceXAI's newest Grok 4.5 model on GDPval+: real professional deliverables (from a variety of economic sectors and occupations), graded against expert-authored rubrics. Grok 4.5: 29%. GPT 5.5: 22%. Claude Opus 4.8: 21%.

English

711

1.3K

4.6K

1.7M

Henry Kiss Ehrenberg retweetledi

vincent sunn chen@vincentsunnchen·6d

Fable 5 tops the Senior SWE-bench leaderboard and exhibits "situational awareness" in 3x more trials than Opus 4.8 Higher-capability models are becoming more benchmark-aware, and we are hardening our eval protocols in response

Claude Fable 5 results are in! It's the new Senior SWE-bench leader at 27.9%, 3 pts above Opus 4.8 (the previous #1). Fable 5 excels at open-ended feature tasks, improving 35% over Opus 4.8. It's also more expensive: 8x more output tokens than GPT-5.5 on avg. And we observed some capability jumps that we didn't fully expect.

English

3

21

1.5K

Henry Kiss Ehrenberg@henryehrenberg·7 Tem

And last update: Senior SWE-bench is now also on Harbor Hub (@harborframework)! You can run `harbor run -d snorkel-ai/senior-swe-bench-v2026.06` out of the box. Full blog post: senior-swe-bench.snorkel.ai/blog/2026-07-0…

English

3

15

884

Henry Kiss Ehrenberg@henryehrenberg·7 Tem

Fable 5 appears to be much more situationally aware than previous models. We found signals that it recognized it was being benchmarked 3x more often than Opus 4.8. This tends to correspond with the agent exploring vectors for solution leakage, so we've made a couple changes to further mitigate opportunities for reward hacking (including a stricter internet allowlist). We recommend all benchmark makers review their existing controls and analysis tools as more eval-aware models are released.

English

4

8

44

2.7K

Henry Kiss Ehrenberg@henryehrenberg·7 Tem

Claude Fable 5 results are in! It's the new Senior SWE-bench leader at 27.9%, 3 pts above Opus 4.8 (the previous #1). Fable 5 excels at open-ended feature tasks, improving 35% over Opus 4.8. It's also more expensive: 8x more output tokens than GPT-5.5 on avg. And we observed some capability jumps that we didn't fully expect.

English

5

7

46

8.8K

Henry Kiss Ehrenberg@henryehrenberg·3 Tem

@JeffreyFC1225 @harborframework We actually did some experimentation with this early on! We expected some preference but actually find anything significant when testing across GPT and Claude models (as solvers and judges). Agree this would be a cool blog, can look into refreshing the results.

English

0

1

53

Jeffrey Yang Fan Chiang@JeffreyFC1225·3 Tem

@henryehrenberg @harborframework Very nice work. I wonder if the scores change when the judge changes from one provider to another. It will be cool to see if the judge prefers their own answer if they are the same model (family). (might be a cool blog post)

English

0

1

53

Henry Kiss Ehrenberg@henryehrenberg·1 Tem

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns. Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions. We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that. Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

English

14

60

239

79.9K

Henry Kiss Ehrenberg@henryehrenberg·3 Tem

@MsShannon112 @harborframework We’re running it now!

English

33

Man@MsShannon112·3 Tem

@henryehrenberg @harborframework WHen will you benchmark Fable?

English

0

46

Henry Kiss Ehrenberg@henryehrenberg·3 Tem

@nick_kango @vincentsunnchen @fredsala @karthik_r_n @GOrlanski @ivanleomk Couldn't say no to that! Shoot me a DM.

English

0

42

Nick@nick_kango·2 Tem

@henryehrenberg @vincentsunnchen @fredsala @karthik_r_n @GOrlanski Congrats:) wanna collaborate to publish it on Kaggle with more models? cc @ivanleomk

English

0

3

55

Henry Kiss Ehrenberg@henryehrenberg·2 Tem

tl;dr the leaderboard includes all models' best performing (or only) reasoning/effort level. Kimi 2.6 only has one available reasoning level (marked as default). Gemini 3.5 Flash performed significantly worse at its highest reasoning level (high) than medium, so we reported medium. Every other model scored best at its highest available effort level.

English

3

146

jehrjd@jehrjd45963·2 Tem

@henryehrenberg @harborframework using default effort for kimi 2.6 and medium effort for gemini 3.5 flash while using max and xhigh effort for everything else is just so silly. what's even the point of comparing them if you're not going to put them all at their max reasoning effort?

English

0

193

Henry Kiss Ehrenberg retweetledi

Karthik Narasimhan@karthik_r_n·2 Tem

When we first conceptualized SWE-bench in the summer of 2023, the driving question was "What do software engineers really do and how do we measure LLMs on those tasks?" (this was back when HumanEval was the standard coding benchmark, testing models on simply code generation) Fast forward to now and both the models and the SWE role have leveled up significantly -- software engineers now spend a lot more time on understanding requirements, tasteful design and validating solutions than writing actual code. And Senior SWE-bench is designed to push those limits even further by testing modern SWE agents on tackling such underspecified problems and evaluating them for both correctness and taste. Be sure to read @henryehrenberg 's excellent blog articles (senior-swe-bench.snorkel.ai/blog) on the various design choices as well as an analysis of the first round of results!

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns. Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions. We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that. Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

English

2

13

43

5K

Henry Kiss Ehrenberg@henryehrenberg·2 Tem

@lik_ai_ @harborframework Planning to add soon! Just mapping a couple last providers.

English

5

393

Kai@lik_ai_·2 Tem

@henryehrenberg @harborframework Is it possible to get the cost per task in the bench?

English

0

2

443

Henry Kiss Ehrenberg@henryehrenberg·1 Tem

@OfirPress Just used it for Senior SWE-Bench!

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns. Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions. We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that. Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

English

All evaluations were run using mini-swe-agent as harness, and through official model APIs. Evaluation settings for each model are also shown on our site. Tracking cybersecurity capabilities of AI agents is an important area of work that we will continue investing effort in. We’re grateful to CyberGym’s seminal work that we built on top of (adding new vulnerabilities as well as the patching task). We are also grateful to the ARVO project for creating infrastructure that we used in our evaluation. Going forward, we plan to both expand the benchmark, and also partner with labs via trusted access programs to test models with and without guardrails to distinguish actual model capability and what is available via API to the public. See full results here: vals.ai/benchmarks/cyb…

0

10

215

Ofir Press@OfirPress·1 Tem

Our mini-swe-agent is becoming a standard harness for running benchmarks across industry and academia, because it's easy to run & extend while achieving similar or better performance than Claude Code and Codex on benchmarks.

Vals AI@ValsAI

English

7

2

44

9K

Henry Kiss Ehrenberg retweetledi

Fred Sala@fredsala·1 Tem

New benchmark I’m excited about: Senior SWE-Bench. Simple idea: if coding agents are to act like senior engineers, we should evaluate them on senior-engineer work: underspecified, messy bugs, investigations, maintainability, and code taste. Awesome work led by @henryehrenberg!

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns. Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions. We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that. Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

English

14

47

2.3K

Henry Kiss Ehrenberg retweetledi

Braden Hancock@bradenjhancock·1 Tem

New gold standard benchmark for measuring agentic coding abilities just dropped: Senior SWE-Bench. Three things I particularly like about this benchmark: 1. It focuses on the next frontier for coding agents: not complete this line, complete this file, or even complete this PR. The instructions are a high-level functionality request and solutions require a level of architect-level thinking, clarifying requirements and making tasteful decisions. 2. Innovation in how to verify solutions. The reason benchmark tasks tend to be so prescriptive is because you need answers in a super specific format to be able to verify reliably. But overly prescriptive task descriptions are out of distribution with how these agents are actually being used. I expect to see more validation agents like this moving forward. 3. Just really really good eval hygiene. Hidden test set, monitoring for cheating, task diversity analysis, failure mode analysis, compatible with @harborframework on day 0, etc. Check out both blogs (one on the benchmark design, the other analyzing results) -- clear craftsmanship. Not surprising coming from the OG SWE-Bench team and @SnorkelAI. Nice work @henryehrenberg, @vincentsunnchen, @karthik_r_n, @gorlanski, and @fredsala! senior-swe-bench.snorkel.ai/blog/2026-06-1… senior-swe-bench.snorkel.ai/blog/2026-06-3…