Our small English read-speech test measured 1.23% word error rate on clean audio and 4.48% on harder recordings. The clean result corresponds to a 98.77% word score, using 100 minus word error rate. It is a useful result for this sample, and it does not establish 99%+ accuracy across podcasts.
We ran 48 transcription requests across four conditions, retained every response, and published the references and scoring code. This article explains what we measured, what went wrong, and how to repeat the experiment.
Our earlier copy described transcription as achieving 99%+ word-level accuracy on clean audio without a supporting repository benchmark. We have replaced that general claim with qualified copy and this evidence. For the product workflow, see how Podtyper transcribes podcasts.
The results
The run took place on October 18, 2026. Each condition contains 12 requests. There were no failed calls, inference retries, or discarded outputs.
| Condition | Audio seconds | Reference words | Substitutions | Deletions | Insertions | WER | Word score |
|---|---|---|---|---|---|---|---|
| Clean recordings | 92.515 | 244 | 3 | 0 | 0 | 1.23% | 98.77% |
| Harder recordings (test-other) | 81.455 | 223 | 6 | 3 | 1 | 4.48% | 95.52% |
| Clean recordings, second run | 92.515 | 244 | 3 | 0 | 0 | 1.23% | 98.77% |
| Clean recordings with added noise | 92.515 | 244 | 3 | 1 | 0 | 1.64% | 98.36% |
There are 24 unique recordings, 24 speakers, 467 unique reference words, and 173.97 seconds of original audio. The clean repeat and noisy versions reuse the original clean recordings. All requests together process 359 seconds; that is not 359 seconds of independent material. Clip lengths range from 1.93 to 23.74 seconds.
The clean repeat produced the same normalized words and edit counts. Some punctuation differed. That checks consistency on these clips during this run; it does not guarantee that a future model response will be identical.
Audio and sample selection
We used LibriSpeech, published through OpenSLR: English read speech derived from audiobooks, distributed under CC BY 4.0. Credit goes to Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. We downloaded selected examples from its Hugging Face distribution.
Before any transcription request, we fixed 12 evenly spaced row indices from each test partition: floor(i × (N − 1) / 11), for i from 0 to 11. N is 2,620 for clean and 2,939 for other. We did not screen clips for favorable recognition results. IDs, reference texts, durations, and audio SHA256 hashes are frozen in the manifest.
The noisy condition adds deterministic Gaussian white noise to the clean clips at a 10 dB signal-to-noise ratio measured across each whole clip. Its seed, scaling, and PCM16 WAV encoding are documented in the kit. Synthetic noise is one controlled stress test; it does not reproduce music, echo, clipping, or overlapping voices.
Model and request settings
We tested grok-voice-transcribe-2.0 through the xAI batch speech-to-text API, using the model and basic request settings currently configured in Podtyper’s worker. Requests explicitly set the model and diarize=false, with options before the audio file. They used automatic language detection, no keyterm hints, and no cleanup or chunking. Other provider options were left at their defaults.
This is a direct evaluation of the STT response text. The live worker can also supply episode-specific vocabulary hints. We did not measure podcast link resolution, downloads, final application formatting, speaker labels, timestamps, AI summaries, or exports. Calling these results an end-to-end podcast accuracy benchmark would overstate the experiment.
How we calculated word error rate
WER = (substitutions + deletions + insertions) / reference words. A substitution replaces a reference word, a deletion omits one, and an insertion adds one. We align words using minimum edit distance and sum the edits and reference words across the condition. We do not average clip percentages, which would give a seven-word clip the same influence as a 62-word clip.
The scorer normalizes Unicode, lowercases text, standardizes curly apostrophes, replaces punctuation with spaces, and splits on whitespace. It preserves internal apostrophes and accents. It does not expand numbers or contractions, change spelling, or correct text using the reference.
The word score in the table is simply 100 × (1 − WER). It is not model confidence, semantic accuracy, or the percentage of quotations safe to publish. Insertions can even make WER exceed 100%.
What the errors looked like
Some counted errors were spelling or formatting differences:
- In clean clip
3729-6852-0006, the reference spells a name SILVIA; the response spells it Sylvia. - In
4992-23283-0002, MISSUS becomes Mrs., counted as a substitution after punctuation normalization. - In
121-123852-0004, PIERC'D becomes pierced, another substitution under the fixed rules.
We kept those errors in the reported score. Expanding abbreviations or modernizing the reference after seeing the output would change the scoring rules.
Other misses affect meaning. In harder clip 2609-156975-0036, destinies became death’s needs. In the noisy version of 5142-33396-0054, the response omitted for. A readable transcript can still contain errors that matter to a quote or interpretation.
What this does and does not show
The model recognized most reference words in this sample, and harder recordings had a higher observed error rate. Adding noise introduced one additional edit across the clean set. These are descriptive results, not estimates of the average error rate across all podcasts or proof of a statistically significant difference between conditions.
The sample is small, chosen by a fixed convenience rule rather than random sampling. One additional edit changes the clean score by about 0.41 percentage points. We did not test conversational podcasts, overlapping speech, other languages, specialist terminology, long episodes, or competing services. Public audiobook material may also have appeared in model training; we cannot rule that out.
For a creator, the practical takeaway is to review names, technical terms, and quoted passages against the audio. Our podcast transcription guide explains where transcript review fits in the workflow.
Download and reproduce
The evidence is available without an account:
- Sample IDs, references, durations, and audio hashes.
- Every model response, timestamp, latency, and score.
- Reproduction kit: preparation script, runner, scorer, tests, and instructions.
Unzip the kit and follow benchmarks/README.md. To recalculate the archived scores without an API key or new provider charges:
npm install
npm run benchmark:score
npm test
To run inference again, install the pinned Python requirements, prepare the audio, and provide your own xAI API key through the environment. Use a new output filename to preserve the archived evidence. The kit verifies audio hashes and records failed calls; it does not retry inference or select the best result. Live calls incur normal provider charges.
Reproducibility here means that the inputs, scoring rules, and request configuration are inspectable and repeatable. Provider behavior can change even when a model name stays fixed. A broader podcast dataset is needed before making broader podcast accuracy claims.