When we first reported a podcast word error rate near 10%, the natural reaction was: that sounds like a lot. Does it mean one word in ten is wrong? Does it mean the transcript is only 90% useful?
The first interpretation is a rough intuition, with some qualifications. The second does not follow from the metric.
Word error rate is a count of text edits needed to match a reference transcript. It is useful because it is explicit and repeatable. It is limited because an extra “um,” a wrong surname and a reversed number can each contribute a word error while having very different consequences.
We make Podtyper. The measured figures below come from our sixteen-recording podcast benchmark; the short sentence example is invented to explain the arithmetic.
Three kinds of word error
The standard calculation is:
WER = (substitutions + deletions + insertions) / reference words
A substitution replaces a reference word with another word. A deletion leaves a reference word out. An insertion adds a word that is not in the reference. A word-alignment algorithm finds a minimum-cost sequence of these edits. Sometimes several equally good alignments exist, so the individual error labels are less stable than the total.
Here is a deliberately small example:
| Position | Reference | Transcript | Error |
|---|---|---|---|
| 1 | we | we | — |
| 2 | met | meet | Substitution |
| 3 | alex | alex | — |
| 4 | at | at | — |
| 5 | the | — | Deletion |
| 6 | old | old | — |
| 7 | station | station | — |
| 8 | on | on | — |
| — | — | next | Insertion |
| 9 | monday | monday | — |
| 10 | morning | morning | — |
There are ten reference words and three edits, so WER is 3/10, or 30%. This is an illustrative sentence, not output from a tested app.
Insertions explain why “one wrong word in ten” is not an exact definition. Some errors are extra words, rather than incorrect replacements for a reference word. With enough insertions, WER can even exceed 100%.
A reference is an editorial choice too
A human transcript may preserve every hesitation or edit speech into readable prose. Those are different targets.
A recognizer that drops fillers can disagree with a verbatim reference even when it preserves the substantive words. A recognizer that keeps repetitions can disagree with an edited reference because it retained speech the editor removed.
That is why a score needs a reference style attached to it. “10% WER” without the reference, normalization rules and audio conditions leaves out much of the information needed to interpret it.
In our metadata-hint podcast condition, strict WER was 10.18% against verbatim references and 11.43% against nonverbatim references. The output did not change between those two calculations. The target did.
How the same output went from 10.18% to 8.29%
Before aligning words, a scorer usually normalizes text. Even our original strict scoring view applies fixed preprocessing. The supplementary English view accepts more equivalences using a pinned version of Whisper’s English normalizer.
Consider a number written as words in one transcript and digits in another. A fixed number-normalization rule can make those forms agree. Some contraction and filler handling can also change which tokens reach the alignment.
For the original metadata-hint outputs, the exact counts were:
| Verbatim scoring view | Word errors | Reference words | WER |
|---|---|---|---|
| Original strict rules | 19,432 | 190,803 | 10.18% |
| Supplementary English normalizer | 16,385 | 197,585 | 8.29% |
Notice that the denominator changed too. Normalization can alter tokenization and word forms on both sides. It is not simply a process of deleting 3,047 mistakes from an otherwise identical word list.
The normalization audit is useful for separating some written-form differences from recognition disagreement. It is not a semantic accuracy score, and we did not replace the original primary measure after seeing a lower number.
Why names and numbers deserve their own checks
Imagine two transcripts with the same WER. One mostly disagrees about fillers. The other changes a guest’s name, a date and the word “not” in a quotation. The aggregate does not tell you which transcript is safer to publish.
Our metadata experiment makes this concrete. Hints recovered more selected name occurrences, yet also produced more unmatched mentions. A single WER value would not describe that tradeoff well.
For research or publication, separate the questions. How closely does the text match a reference overall? Are the names and numbers correct? Does a proposed quotation match the audio, including who said it? Each needs appropriate evidence. We have measured the first question and a selected vocabulary checklist; we have not independently verified every quotation in the corpus.
How to combine several episodes
Suppose a ten-word excerpt has two errors and a thousand-word excerpt has fifty. Their WER values are 20% and 5%.
An unweighted average is 12.5%. A word-weighted corpus score is 52 errors divided by 1,010 words, about 5.15%. Both calculations are possible, but they answer different questions.
Our corpus score uses the latter approach. We add errors and reference words before dividing, while also showing the result for each recording. That gives long episodes appropriate influence without hiding where individual recordings did poorly.
If you compare two published leaderboards, check their aggregation rules as well as the dataset names. Some weight tasks, durations or datasets differently. The limitations in our podcast benchmark explain why the available results do not establish a shared app ranking.
Is 10% good enough?
It depends on the job and the kinds of errors. This metric alone cannot answer whether a transcript is ready for subtitles, a research quotation or a quick personal read-through.
Use it to compare systems under the same conditions and to track changes using a fixed test. Then inspect the errors that matter for your task. For a quotation, replay the relevant passage. For a technical interview, check names, terms and numbers. If correction time matters, measure correction time rather than deriving it from WER.
The compact results file and reproducible scoring kit retain the counts behind our figures. External evaluators such as Artificial Analysis also publish their own normalization and aggregation methods; read those before carrying a score from one context into another.
A 10% WER is a useful measurement. The useful response is to ask what was counted, what was tested and which errors remain—not to turn it directly into a percentage of trust.