Alex Veremeyenko@alex_verem
Researchers replayed 740 "official" coding-agent benchmark patches on different cloud machines. only 11 out of 140 kept their validity signal.
a team from singapore management university and shanghai jiao tong university audited the three benchmarks everyone cites to measure whether coding agents can optimize code. gso, swe-perf, and swe-fficiency. they took every official reference patch and ran it across four google cloud machine types, three rounds each.
the results are rough. swe-perf kept only 11 out of 140 tasks valid. gso kept 39 out of 102. swe-fficiency held up best at 411 out of 498 but still lost 87 tasks. these are the reference patches the benchmarks themselves say are correct. swap the machine and the optimization signal disappears.
it gets worse. the same eight coding agents rank differently depending on which benchmark you look at. nine out of twenty-eight head-to-head comparisons flip between gso and swe-fficiency. one model jumped five positions. and swe-fficiency's scoring formula lets the worst ten tasks carry 58 to 82 percent of a submission's entire score weight. one bad task can outweigh hundreds of good ones.
the deeper finding is that 384 out of 450 valid tasks are already matched or beaten by at least one public submission. the benchmarks are running out of room to separate strong agents from weak ones.
swe-perf's core problem is that its reference patches cluster near zero runtime change. the median improvement is negative 0.03 percent. at that scale, normal machine noise is bigger than the optimization itself. the benchmark is measuring hardware jitter, not agent capability.
we keep treating leaderboard scores as hard evidence that coding agents are improving. this paper is a reminder that the ruler we're using bends depending on which shelf you put it on.
paper link in the reply.
