Sabitlenmiş Tweet
Datacurve
22 posts
Datacurve
@datacurve
Research and data to advance frontier models.
San Francisco Katılım Şubat 2024
11 Takip Edilen8.9K Takipçiler
View the full family of GPT-5.6 models on our benchmark → deepswe.datacurve.ai
English
Datacurve is at @aiDotEngineer!
Come by our booth in the expo hall to learn more about DeepSWE (and snag some merch).
Plus, join us on Thursday @ 10:30am for our talk.
English
Datacurve retweetledi
We've updated the Artificial Analysis Coding Agent Index, replacing SWE-Bench Pro with Datacurve's DeepSWE benchmark - the swap lifts Codex with GPT-5.5 (xhigh) above Claude Code with Opus 4.8 (max), while the newly released Claude Fable 5 (max) in Claude Code debuts at the top
DeepSWE, built by @datacurve, writes its tasks from scratch rather than adapting them from public GitHub issues or pull requests, so no model has seen the solutions during training. That matters because SWE-Bench Pro, the benchmark it replaces in our Coding Agent Index, had grown gameable, with some models recovering the fix from the repository's commit history instead of solving the task.
The swap reorders the index: Codex with GPT-5.5 (xhigh) rises from 65 to 76, overtaking Claude Code with Opus 4.8 (max) at 73. Claude Code with Fable 5 (max), which enters directly on the refreshed index, leads at 77. SWE-Bench Pro had been flattering some combinations and penalizing others.
More below.
English
Opus 4.8 is a clear efficiency step up over 4.7 on DeepSWE, same-or-better task success with fewer steps and fewer input tokens per task.
Worth noting DeepSWE measures backend/implementation, not frontend.
Hoping to release a deep dive soon.
Datacurve@datacurve
Opus 4.8 is now on DeepSWE. On the default high thinking effort, it scores 6% higher than Opus 4.7 xhigh, while also lowering average cost per task.
English
Full deep dive coming soon.
Check out the full benchmark here → deepswe.datacurve.ai
English
Datacurve retweetledi
DeepSWE reflects what I’m hearing from engineers better than any other benchmark.
They took the hard path to build a good one.
Serena Ge@serenaa_ge
Today we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
English
Datacurve retweetledi
This is the new standard for engineering evals
Serena Ge@serenaa_ge
Today we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
English