We are seeing a sustained statistically significant degradation in Claude Code with Opus 4.7 since last Friday May 22nd at marginlab.ai/trackers/claud…
English
Margins
27 posts
We identified an issue with our Codex test harness that caused a subset of test instances after May 9th to use gpt-5.5-high instead of gpt-5.5-xhigh
This issue exclusively affected our Codex tracker (marginlab.ai/trackers/codex/) and has since been resolved
English
There is more to the story:
1. Agree on the lessons, eval infra is critical
2. Our testing is robust to infra. We re-try any test instance that fails to generate a patch at marginlab.ai/trackers/claud…
3. In spite of (2), we've detected statistical degradation for CC several times
Thariq@trq212
really good post on why agentic coding evals are so noisy and might vary by several percentage points between runs
English
@ThePrimeagen Correct, this is an all-time low since Opus 4.6 launched. Any sustained drop today will tip our tracker into statistically significant degradation territory
English
Margins retweetledi
Claude Opus 4.6 had it's worst benchmark day yesterday
English
Margins retweetledi
@alexatallah We are building an agent harness benchmarking platform at marginlab.ai , we use it internally to run our daily benchmarks, e.g. Claude Code benchmarks (marginlab.ai/trackers/claud…)
Coming soon™
English
Claude Code with Opus 4.6 Degradation Tracker is live at marginlab.ai/trackers/claud…
See historical data (including Opus 4.5) under marginlab.ai/trackers/claud…
Tracker for Codex with gpt-5.3-codex available too!
English
@levelsio Checkout our the trackers at marginlab.ai/trackers/claud…
Daily evals on Claude Code/Codex
English
My secret conspiracy theory about AI companies is they nerf models to save on compute
Then they check X to see if anyone notices it
If yes, give back compute
If not, continue
@levelsio@levelsio
I have never experienced a more dumb Claude Code than today, I have to start coding myself again cause it makes so many Low IQ mistakes, they must be nerfing it now that Opus 4.6 is out or something is up
English
Claude Code continues to show a statistically significant degradation over the past month
marginlab.ai/trackers/claud…
English
the real alignment scare from this week:
a subtle bug in claude code made the model much dumber. software engineers worldwide were significantly less productive for two days
not just one anecdote, many people have been telling me this
it's fixed now apparently
English
Claude Code Degradation Tracker: Seeing a sustained (but not statistically significant) drop in performance over the past few days.
See details here: marginlab.ai/trackers/claud…
English