What Word Error Rate Measures, and Why Speech-to-Text WERs Disagree

The formula: substitutions, deletions and insertions

Word error rate (WER) lines up a system's transcript with a human reference transcript, word by word, and counts three kinds of mistake: substitutions (S, a wrong word), deletions (D, a missing word) and insertions (I, an extra word). The total is divided by the number of words in the reference, N: WER = (S + D + I) / N.¹ In the leaderboard's own example, the reference "the cat sat on the mat" transcribed as "the cat sit on the" has one substitution and one deletion, so WER = 2 / 6, or 33%.¹ The result reads as the average number of errors per reference word, and lower is better.⁸

Two consequences follow from the definition. Insertions count in the numerator but not in N, so a system that adds many words that were never spoken can score above 100%. The jiwer library's documentation gives the extreme case: two words transcribed over silence (an empty reference) score a WER of 2, or 200%, which it uses to test whether models hallucinate on silent audio.⁹ And every word counts the same: "sit" for "sat" is one wrong character but a whole wrong word.⁷ For languages written without spaces between words, such as Mandarin and Japanese, evaluators use the character error rate (CER) instead.⁷

Text normalization decides what counts as an error

Models differ in whether they output punctuation, capital letters, numerals or filler words such as "um", and reference transcripts differ too. Scored raw, "Twenty dollars." against "twenty dollars" would be counted as a mistake. So the Open ASR Leaderboard normalizes both texts before computing WER: it removes punctuation and casing, converts number words to digits ("zero" to "0"), standardizes spelling, drops filler words and treats compounds such as "white paper" and "whitepaper" as equal, in a pipeline that closely follows OpenAI's Whisper normalizer.¹⁵ Hugging Face's audio course notes that normalizing makes WERs "naturally lower", though the model "isn't necessarily better for production".⁷

That makes the normalizer part of the benchmark. The leaderboard switched to an updated English normalizer on June 24, 2026, so WERs from before and after that date aren't strictly comparable.¹ A WER quoted without its normalizer is missing half its definition.

The test set sets the difficulty

WER depends heavily on the audio. The chart above shows one model, Zoom's Scribe v2 Pro, at 1.11% on LibriSpeech clean (5.4 hours of read audiobooks) and 6.08% on AMI (9 hours of spontaneous meeting speech), 7.99% on URGENT24 (speech overlaid with noise and room reverberation) and 12.10% on the private conversational sets.¹ Hugging Face's WER metric card makes the same point across datasets: LibriSpeech results are typically reported in the 1.8 to 3.3 range, TIMIT results in the 8.3 to 20.4 range.⁸

The order of models changes by test set as well. In the table, Modulate's Multilingual has the lowest LibriSpeech clean WER on the whole leaderboard (0.87%) but ranks third on average, while Zoom's Scribe v2 Pro is lowest on Earnings22, AMI, URGENT24 and the private conversational sets.¹ The reference transcripts matter too. On the original Earnings22 set, ElevenLabs' Scribe v2 scores 8.90% and Reson8's Resonant-1 8.79%; on the cleaned 1.9-hour version prepared by Artificial Analysis, which the leaderboard switched to on Aug 21, 2026, the same models score 4.80% and 5.48%.¹² These are different versions of the test set, so the two numbers never belong in the same column.

Averages: which sets, and how they're weighted

A single "average WER" hides two choices: which test sets go in, and how much each one counts. The Open ASR Leaderboard takes a macro-average, so each of its 11 sets counts equally, whether it holds 1.92 hours of audio (URGENT24) or 100 hours (SPGISpeech).¹ A model gets an average only if it has a score on every set; otherwise its average is shown as "NA".⁴ The set list also changes: private data joined the default average on July 24, 2026, and URGENT24 on Oct 9, 2026.¹³

The raw results file shows why that rule matters. It still carries an "avg" field, which the leaderboard's code ignores; for Reson8's Resonant-1 Flash it reads 4.02, the lowest in the file, but the model has no URGENT24 score, so the leaderboard gives it no average and no rank.²⁴ Artificial Analysis averages differently: its AA-WER v2 is weighted by audio duration across about 8 hours from three datasets, with its own AA-AgentTalk set at 50% and cleaned VoxPopuli and Earnings22 sets at 25% each.¹⁰

Harness and decoding settings

The code around a model changes its score. The leaderboard cuts short-form audio into chunks of at most 30 seconds, with a few exceptions, and for its Earnings22 set it transcribes each chunk and joins them before scoring against the full transcript.⁵¹ It runs English evaluations as Docker images on the same hardware through Hugging Face Jobs, to keep the environment fixed.⁶ Decoding matters even for one set of weights: the same SoundsgoodAI Zipformer model averages 6.82% with transducer beam search and 8.38% with greedy CTC decoding.¹

Hosting matters as well. On Artificial Analysis, OpenAI's Whisper Large v3 measures 4.1% through fal.ai, 4.5% through Together AI and 10.1% through Replicate.¹⁰ A WER describes a model run in a particular way, not the weights alone.

Who ran the test: vendor-run and independent numbers

To add a model to the Open ASR Leaderboard, a contributor opens a pull request with evaluation scripts and self-reported metrics; the maintainers re-run the scripts in their own environment before the leaderboard is updated, and evaluate the model on the private sets themselves.⁵⁶ The private sets exist to "address risks of overfitting to public datasets": the leaderboard's paper notes it can't fully rule out that a model was trained on a public test set.¹⁵ A vendor's own WER, measured on its own audio with its own normalizer and average, is a vendor-stated figure; MetaPrompt never ranks it against independent results.

Even two independent leaderboards disagree. Artificial Analysis measures ElevenLabs' Scribe v2 at 2.2% AA-WER and Zoom Scribe V2 Pro at 2.8%; the Open ASR Leaderboard averages them at 5.24% and 4.70%, so the order flips.¹⁰¹ Mistral's Voxtral Small is the open-weights model with the lowest AA-WER (2.8%) on Artificial Analysis and its 24B-2507 version ranks 30th on the Open ASR Leaderboard at 6.38%.¹⁰¹ Neither is wrong: they measure different audio with different averages, which is why this page shows one benchmark per table.

What WER doesn't capture

Because punctuation and casing are stripped before scoring, a leaderboard WER says nothing about how readable the formatted transcript is.¹⁷ It compares word strings only, so it doesn't score who said each word (speaker labels) or when. It treats a harmless slip and a meaning-changing error the same, and, as Hugging Face's metric card puts it, "provides no details on the nature of translation errors".⁸ Speed is a separate measurement: the Open ASR Leaderboard reports the inverse real-time factor (RTFx, seconds of audio transcribed per second of compute) from the same run, and Artificial Analysis reports a speed factor and a price per 1,000 minutes alongside AA-WER.¹¹⁰