---
title: "Podtyper Transcription Accuracy: 16 Hours of Real Podcasts"
seoTitle: "Podcast Transcription Accuracy: 16-Hour Benchmark"
description: "16.15 hours of real podcasts, human references, strict and normalized word error rates, names, vocabulary hints, and controlled Grok STT settings tests."
date: "2026-10-18"
lastModified: "2026-10-27"
author: "Berke Atac"
authorTitle: "Founder, Podtyper"
twitterHandle: "@berkeatac"
category: "Research"
tags: ["transcription accuracy", "word error rate", "benchmark", "podcast transcription"]
featured: true
---

**We tested all 16 recordings in the public Rev16 podcast benchmark: 16.15 hours across 12 shows, using complete audio files rather than short audiobook clips.** On this set, the word error rate against human nonverbatim references was **11.37% without context** and **11.43% with metadata vocabulary hints**.

The [follow-up scoring audit and settings experiment](#follow-up-scoring-audit-and-grok-parameter-tests) reports 8.29% supplementary English-normalized verbatim WER on the original metadata outputs, alongside the unchanged strict scores. Normalization changes scoring rules; it does not establish semantic accuracy.

We also checked selected names, terms and brands occurrence by occurrence. This separates a useful overall score from errors in a guest's name or a specialist term. Results vary by show; review names, numbers and important quotations against the original audio before publishing.

The October 27, 2026 run made **35 requests**: each recording with and without metadata hints, plus three title-only requests to examine description hints. All outputs were retained, with **0 failed calls**, no inference retries and no best-result selection. This updates the evidence behind [how Podtyper transcribes podcasts](/podcast-transcription). The earlier 48-request audiobook study is preserved below.

## Full-episode podcast results

The primary reference is Rev's human nonverbatim transcript: lightly edited for readability. We also report the human verbatim reference, which retains fillers, repetitions and stutters. Both reference styles and the scoring code were frozen before inference. We use the same primary reference style for every episode, rather than selecting the better score episode by episode.

| Reference and audio set | Without context WER | With metadata hints WER | Reference words per condition |
|---|---:|---:|---:|
| Nonverbatim, all recordings (primary) | 11.37% | 11.43% | 177,247 |
| Verbatim, all recordings | 10.19% | 10.18% | 190,803 |
| Nonverbatim, conversational subset | 12.01% | 12.06% | 160,796 |

There are 12 conversational recordings. The full-set result also retains one trailer, one fiction reading, one sermon and one solo advice episode. Those four are not counted as conversational evidence. Recordings range from 3.1 to 131.9 minutes. Paired requests reuse the same audio: this is **16.15 unique hours**, not 35 independent recordings.

Across the 16 paired recordings, metadata changed primary WER by **+0.060 percentage points**. A 95% show-cluster bootstrap interval is **[-0.030, +0.203] percentage points**, from 10,000 resamples of 12 shows, seed 20261027. Negative changes favor context. The interval includes zero, so this run does not establish a consistent overall WER improvement.

## Names and terminology

The checklist was frozen before transcription. Each reference occurrence must align to the complete, correctly spelled phrase in the output after the same fixed normalization. Recognizing a name once does not repair another missed occurrence. The following uses the primary nonverbatim reference.

| Selected vocabulary | Reference occurrences | Correct without context | Correct with metadata hints |
|---|---:|---:|---:|
| People's names | 57 | 31 (54.39%) | 43 (75.44%) |
| Domain terms and named topics | 110 | 101 (91.82%) | 103 (93.64%) |
| Brands and platforms | 257 | 251 (97.67%) | 252 (98.05%) |

Examples on the checklist include Ahyiana, Rebekah Taussig, Gidi Chamdi, Karen Kilgariff, Georgia Hardstark, autocross, GTR, Health at Every Size and intuitive eating. The downloadable scores include each term's denominator, matched occurrences, output occurrences, precision and whether it was supplied as a hint. Some checklist terms never occur in a reference and contribute no recall denominator. First names and full names can overlap; these are checklist counts, not independent people or an exhaustive named-entity benchmark.

The recall gain has a tradeoff: aligned precision for selected names fell from **91.18% without context to 75.44% with metadata hints**. There were 3 unmatched checklist-name output occurrences without context and 14 with hints. These are disagreements with the reference alignment, not independently adjudicated hallucinations. Supplying a spelling can encourage both correct and misplaced uses, so recall alone would give an incomplete picture.

Spelling variants, contractions and number formatting can count as errors. We did not change reference spellings or accept new variants after seeing responses. Both human reference styles are available to inspect. This audit checks agreement with those references; it is not a new human adjudication of every quote or name.

## How Podtyper uses context before transcription

When episode metadata is available, Podtyper extracts vocabulary from the show title, episode title, description, author and guest metadata, and sends the resulting terms to the speech recognizer **before** it transcribes the audio. This gives it spellings for names, organizations and episode-specific vocabulary. Descriptions can contain names that the title omits.

This experiment uses the worker's actual vocabulary extractor and request builder. Every episode has show and episode titles. For three episodes, we also supplied short descriptions curated from public episode notes, with their sources frozen before inference. The other 13 descriptions remain missing. This tests recognition with available context; it does not test whether the live metadata resolver retrieves those notes successfully.

We never supplied the reference transcript, a reference-derived keyword list or corrections from an earlier output as transcription hints. The full keyword lists are in the request plan. Hints can help a difficult spelling, do nothing, or bias recognition toward the wrong word; the result should determine the claim.

### Description hints compared with titles alone

These three extra requests compare title-only vocabulary against titles plus descriptions, using the same full audio. This is a small exploratory comparison with one response per condition.

| Episode | Titles only WER | Titles plus description WER |
|---|---:|---:|
| Podcasts in Color (10) | 8.41% | 8.51% |
| Pod Save America (17) | 9.22% | 9.21% |
| Food Psych (23) | 7.13% | 7.23% |

These results do not isolate a guaranteed causal benefit from descriptions: provider nondeterminism remains, and only three sourced descriptions were tested. The public scores also expose the names and terms in each of these outputs.

## Results for every recording

All entries in the pinned Rev16 catalog are retained. The table uses the primary nonverbatim reference and reports word error rate, not confidence or a quotation-safety rating.

| ID | Show | Minutes | Format | No context WER | Metadata hints WER |
|---|---|---:|---|---:|---:|
| 3 | The Read | 131.9 | conversation | 13.38% | 13.16% |
| 4 | The Read | 126.0 | conversation | 13.33% | 13.40% |
| 9 | Podcasts in Color | 49.6 | conversation | 8.39% | 8.63% |
| 10 | Podcasts in Color | 36.4 | conversation | 8.89% | 8.51% |
| 11 | Podcasts in Color | 39.7 | solo-advice | 7.90% | 8.18% |
| 14 | Crimetown | 3.1 | trailer | 6.21% | 4.52% |
| 17 | Pod Save America | 58.2 | conversation | 9.18% | 9.21% |
| 18 | Pod Save America | 66.3 | conversation | 9.08% | 9.24% |
| 20 | The New Yorker: The Writer's Voice | 35.8 | fiction-reading | 2.20% | 2.22% |
| 21 | Skidmarks Show | 40.3 | conversation | 15.35% | 15.51% |
| 23 | Food Psych | 65.1 | conversation | 7.17% | 7.23% |
| 24 | My Favorite Murder with Karen Kilgariff and Georgia Hardstark | 107.7 | conversation | 11.28% | 11.19% |
| 26 | Drinkin' Bros. | 75.1 | conversation | 18.90% | 19.47% |
| 27 | Roads From Emmaus | 21.8 | sermon | 4.07% | 4.07% |
| 29 | The Ross Bolen Podcast | 64.1 | conversation | 8.93% | 8.93% |
| 32 | Recovery Elevator | 48.0 | conversation | 15.29% | 15.41% |

## Podcast benchmark method and downloads

We used [Rev16's human references and fixed episode list](https://github.com/revdotcom/speech-datasets/tree/c05ab6fd8b4b627d123c922a22a39e993dd37635/rev16), with full MP3 files from its linked mirror. Credit goes to the original podcast publishers, Rev transcriptionists, [Radford et al. (2023)](https://proceedings.mlr.press/v202/radford23a.html), and, for nonverbatim references, [McNamara et al. (2024)](https://arxiv.org/abs/2412.07937). The reference texts are reproduced under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Audio rights remain with the original publishers; the kit links to upstream audio and does not redistribute it.

The model was `grok-voice-transcribe-2.0` through xAI's batch endpoint, with `diarize=false`, automatic language detection, and no cleanup or chunking. Other provider options stayed at their defaults, matching the worker. Primary condition order alternated by episode; two requests ran at a time. We froze the corpus revisions, audio hashes, reference texts, context, vocabulary checklist, request settings and evaluated source hashes before inference.

WER is the sum of substitutions, deletions and insertions divided by the reference word count. Unicode NFKC, lowercasing, apostrophe normalization and punctuation removal are fixed across both conditions. Numbers and contractions are not expanded. We use minimum-edit alignment with RapidFuzz, sum edit and word counts across recordings, and report per-episode scores. We never average episode percentages. The original audiobook scorer uses a different alignment tie rule; minimum total edit distance and the normalization policy are unchanged.

Download the evidence without an account:

- [Podcast sample manifest, references, audio hashes and attribution](/benchmarks/rev16-manifest.json).
- [Frozen request plan and actual submitted keywords](/benchmarks/rev16-plan.json).
- [Pre-run context, description sources and vocabulary checklist](/benchmarks/rev16-context.json).
- [Scores for every episode, vocabulary occurrence counts and paired comparisons](/benchmarks/rev16-results-scores.json).
- [Complete podcast reproduction kit, including every raw provider response](/benchmarks/podtyper-podcast-benchmark-kit.zip).

The kit includes a README, preparation script, runner, scorer, tests and exact worker code used for requests. Recalculate the archived results offline, without an API key or provider charges:

```sh
python3 -m venv benchmarks/.venv
benchmarks/.venv/bin/pip install -r scripts/benchmark/requirements.txt
benchmarks/.venv/bin/python scripts/benchmark/score.py
benchmarks/.venv/bin/python -m unittest discover -s scripts/benchmark -p '*_test.py'
```

Installing dependencies needs network access; subsequent rescoring is offline. For fresh inference, follow the kit's README and use a new output filename. Live requests incur provider charges. A future response may differ even with the same model name.

## What the podcast study establishes

This is substantially more relevant evidence for long English podcasts than the earlier short read-speech test. It measures whole episodes, conversational formats, selected names and vocabulary, and the actual context-hint mechanism.

Rev16 is still a small, older public corpus. Its material may have appeared in model training. It does not represent every show, accent, language or specialist domain. Conversational recordings contain interruptions, but this run has no separately adjudicated overlap intervals or overlap-only score. It does not establish accuracy on simultaneous speakers. We did not evaluate speaker attribution, timestamps, caption reuse, summaries, URL resolution or exports.

The next stronger study should add newly recorded, held-out conversations across more independent shows; independent human review of references; and annotated overlap, names, technical terms and publishable quotations. The current results support sample-specific WER and vocabulary findings. Important quotations still need checking against the audio; [our podcast transcription guide](/blog/how-to-transcribe-a-podcast) explains where that review fits.

{/* STT_SETTINGS_AUDIT_START */}
## Follow-up: scoring audit and Grok parameter tests

We investigated whether the original error rate reflected missed speech, transcription style, or request settings. On the same original metadata-context outputs, **strict verbatim WER was 10.18% and supplementary English-normalized verbatim WER was 8.29%**. The nonverbatim figures were 11.43% strict and 10.79% normalized.

The supplementary view uses the independently published [Whisper English text normalizer](https://github.com/openai/whisper/tree/86098128c0b4f24f0e2aa2994de830614b474227/whisper/normalizers), pinned with its license and source hashes. It standardizes contractions, numbers, spelling variants and a short filler list, and removes bracketed annotations and diacritics. It changes scoring equivalences, not the transcript. It is **not semantic error rate**, and names continue to receive the strict spelling checks above. The original strict scores remain unchanged.

Edited human references can penalize spoken phrases a recognizer correctly preserves. Conversely, removing fillers from the recognizer output creates deletions against a verbatim reference. Reporting both reference styles and the disclosed normalization policies makes that tradeoff visible; we do not choose the better reference per episode.

### Settings tested

Using the [documented xAI batch controls](https://docs.x.ai/developers/model-capabilities/audio/speech-to-text), we tested explicit English, English number formatting, filler preservation, VAD thresholds of 0.1 and 0, and combinations. Three whole recordings from different shows were used for development. Each had eight configurations plus an independent default repeat: 27 requests in total. Metadata keywords stayed fixed.

| Development setting | Strict verbatim WER | English-normalized verbatim WER |
|---|---:|---:|
| english-formatted | 13.25% | 10.74% |
| defaults-repeat | 13.46% | 10.91% |
| defaults | 13.19% | 10.74% |
| english-formatted-fillers-vad-low | 12.26% | 10.62% |
| fillers | 12.23% | 10.65% |
| fillers-vad-low | 12.07% | 10.56% |
| english | 13.33% | 10.73% |
| vad-off | 13.08% | 10.52% |
| vad-low | 13.09% | 10.59% |

The selected automatic-language candidate was `vad_threshold=0`, which disables the speech-activity gate. English-only settings could not become a global default for a multilingual product. We froze the selection before a second stage of 32 requests comparing the candidate with fresh defaults across all 16 recordings. Every response was retained, with no inference retries or best-output selection.

The primary confirmation set contains 11 recordings from 9 shows excluded from parameter development. All Rev16 recordings had already been evaluated in the initial study; this is **show-disjoint parameter confirmation, not a fresh unseen audio corpus**.

| Confirmation set and score | Fresh defaults | VAD disabled |
|---|---:|---:|
| 11 validation recordings, English-normalized verbatim | 8.02% | 7.99% |
| 11 validation recordings, Strict verbatim | 9.66% | 9.62% |
| 11 validation recordings, Strict nonverbatim | 10.73% | 10.67% |
| All 16 recordings, English-normalized verbatim | 8.30% | 8.29% |
| All 16 recordings, Strict verbatim | 10.19% | 10.20% |
| All 16 recordings, Strict nonverbatim | 11.41% | 11.40% |

On the primary validation set the normalized WER difference was **-0.037 percentage points**, with a 95% show-cluster bootstrap interval of **[-0.089, +0.043]**. Negative differences favor the candidate. The gate required at least a 0.2 percentage point gain, an interval wholly below zero, and no selected-name F1 loss exceeding 0.02. The candidate did not pass the predefined confirmation gate, so we retained the existing defaults. A small favorable result on development recordings was insufficient to justify a general improvement claim.

Filler preservation can improve agreement with a verbatim reference without improving substantive recognition. Number formatting can change text matching while preserving the same spoken value. Lowering VAD can recover quiet speech and can also transcribe background noise. We keep those effects separate from claims about names, overlap, or quotation safety.

Download the follow-up evidence:

- [Original-run normalization audit and flagged edit examples](/benchmarks/rev16-normalization-audit.json).
- [Development scores for every setting and episode](/benchmarks/settings-dev-results-scores.json).
- [Frozen confirmation plan and submitted parameters](/benchmarks/settings-confirmation-plan.json).
- [Confirmation scores, vocabulary checks and paired uncertainty](/benchmarks/settings-confirmation-results-scores.json).
- [Decision against the predefined promotion gate](/benchmarks/settings-recommendation.json).
- [Listening review: 48 flagged excerpts with source-audio links](/benchmarks/benchmark-listening-review.html).
- [Complete settings audit kit: raw responses, evaluated code, scorer, protocols and licenses](/benchmarks/podtyper-stt-settings-kit.zip).

The listening queue is unreviewed and intended for checking disagreements against the audio. It does not assume the human reference or the model is always correct. Playback accesses upstream public recordings; the kit does not redistribute audio.
{/* STT_SETTINGS_AUDIT_END */}

## Earlier audiobook study: October 18, 2026

The following historical study used short LibriSpeech read-speech clips without vocabulary hints. Its figures are preserved separately from the newer podcast study.

### The results

The run took place on October 18, 2026. Each condition contains 12 requests. There were no failed calls, inference retries, or discarded outputs.

| Condition | Audio seconds | Reference words | Substitutions | Deletions | Insertions | WER | Word score |
|---|---:|---:|---:|---:|---:|---:|---:|
| Clean recordings | 92.515 | 244 | 3 | 0 | 0 | 1.23% | 98.77% |
| Harder recordings (test-other) | 81.455 | 223 | 6 | 3 | 1 | 4.48% | 95.52% |
| Clean recordings, second run | 92.515 | 244 | 3 | 0 | 0 | 1.23% | 98.77% |
| Clean recordings with added noise | 92.515 | 244 | 3 | 1 | 0 | 1.64% | 98.36% |

There are **24 unique recordings, 24 speakers, 467 unique reference words, and 173.97 seconds of original audio**. The clean repeat and noisy versions reuse the original clean recordings. All requests together process 359 seconds; that is not 359 seconds of independent material. Clip lengths range from 1.93 to 23.74 seconds.

The clean repeat produced the same normalized words and edit counts. Some punctuation differed. That checks consistency on these clips during this run; it does not guarantee that a future model response will be identical.

### Audio and sample selection

We used [LibriSpeech, published through OpenSLR](https://www.openslr.org/12): English read speech derived from audiobooks, distributed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Credit goes to Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. We downloaded selected examples from its [Hugging Face distribution](https://huggingface.co/datasets/openslr/librispeech_asr).

Before any transcription request, we fixed 12 evenly spaced row indices from each test partition: floor(i × (N − 1) / 11), for i from 0 to 11. N is 2,620 for clean and 2,939 for other. We did not screen clips for favorable recognition results. IDs, reference texts, durations, and audio SHA256 hashes are frozen in the manifest.

The noisy condition adds deterministic Gaussian white noise to the clean clips at a 10 dB signal-to-noise ratio measured across each whole clip. Its seed, scaling, and PCM16 WAV encoding are documented in the kit. Synthetic noise is one controlled stress test; it does not reproduce music, echo, clipping, or overlapping voices.

### Model and request settings

We tested `grok-voice-transcribe-2.0` through the [xAI batch speech-to-text API](https://docs.x.ai/developers/model-capabilities/audio/speech-to-text), using the model and basic request settings currently configured in Podtyper’s worker. Requests explicitly set the model and `diarize=false`, with options before the audio file. They used automatic language detection, no keyterm hints, and no cleanup or chunking. Other provider options were left at their defaults.

This is a direct evaluation of the STT response text. The live worker can also supply episode-specific vocabulary hints. We did not measure podcast link resolution, downloads, final application formatting, speaker labels, timestamps, AI summaries, or exports. Calling these results an end-to-end podcast accuracy benchmark would overstate the experiment.

### How we calculated word error rate

**WER = (substitutions + deletions + insertions) / reference words.** A substitution replaces a reference word, a deletion omits one, and an insertion adds one. We align words using minimum edit distance and sum the edits and reference words across the condition. We do not average clip percentages, which would give a seven-word clip the same influence as a 62-word clip.

The scorer normalizes Unicode, lowercases text, standardizes curly apostrophes, replaces punctuation with spaces, and splits on whitespace. It preserves internal apostrophes and accents. It does not expand numbers or contractions, change spelling, or correct text using the reference.

The word score in the table is simply 100 × (1 − WER). It is not model confidence, semantic accuracy, or the percentage of quotations safe to publish. Insertions can even make WER exceed 100%.

### What the errors looked like

Some counted errors were spelling or formatting differences:

- In clean clip `3729-6852-0006`, the reference spells a name **SILVIA**; the response spells it **Sylvia**.
- In `4992-23283-0002`, **MISSUS** becomes **Mrs.**, counted as a substitution after punctuation normalization.
- In `121-123852-0004`, **PIERC'D** becomes **pierced**, another substitution under the fixed rules.

We kept those errors in the reported score. Expanding abbreviations or modernizing the reference after seeing the output would change the scoring rules.

Other misses affect meaning. In harder clip `2609-156975-0036`, **destinies** became **death’s needs**. In the noisy version of `5142-33396-0054`, the response omitted **for**. A readable transcript can still contain errors that matter to a quote or interpretation.

### What this does and does not show

The model recognized most reference words in this sample, and harder recordings had a higher observed error rate. Adding noise introduced one additional edit across the clean set. These are descriptive results, not estimates of the average error rate across all podcasts or proof of a statistically significant difference between conditions.

The sample is small, chosen by a fixed convenience rule rather than random sampling. One additional edit changes the clean score by about 0.41 percentage points. We did not test conversational podcasts, overlapping speech, other languages, specialist terminology, long episodes, or competing services. Public audiobook material may also have appeared in model training; we cannot rule that out.

For a creator, the practical takeaway is to review names, technical terms, and quoted passages against the audio. Our [podcast transcription guide](/blog/how-to-transcribe-a-podcast) explains where transcript review fits in the workflow.

### Download and reproduce

The evidence is available without an account:

- [Sample IDs, references, durations, and audio hashes](/benchmarks/librispeech-manifest.json).
- [Every model response, timestamp, latency, and score](/benchmarks/librispeech-results.json).
- [Reproduction kit: preparation script, runner, scorer, tests, and instructions](/benchmarks/podtyper-benchmark-kit.zip).

Unzip the kit and follow `benchmarks/README.md`. To recalculate the archived scores without an API key or new provider charges:

```sh
npm install
npm run benchmark:score
npm test
```

To run inference again, install the pinned Python requirements, prepare the audio, and provide your own xAI API key through the environment. Use a new output filename to preserve the archived evidence. The kit verifies audio hashes and records failed calls; it does not retry inference or select the best result. Live calls incur normal provider charges.

Reproducibility here means that the inputs, scoring rules, and request configuration are inspectable and repeatable. Provider behavior can change even when a model name stays fixed. See the newer full-episode podcast study above for the expanded evidence and its remaining limits.
