Back to blog
Research9 min readOctober 18, 2026

Podtyper on Real Podcasts: What Our 16-Hour Test Found

We tested 16 complete podcast recordings, checked names and terms, and published strict and normalized word error rates. Here is what the results support.

Berke Atac

Founder, Podtyper

A short audiobook clip tells you something about a speech recognizer. It tells you much less about a two-hour podcast with interruptions, unfamiliar names and a long stretch of casual conversation.

Our earlier accuracy test used short English audiobook clips. That was a reasonable starting point, but too narrow a basis for describing podcast performance. So we ran the recognizer on all 16 complete recordings in Rev16: 16.15 hours from 12 shows, scored against human transcripts.

With episode metadata supplied before recognition, the result was 11.43% word error rate against edited, nonverbatim references. Against verbatim references, it was 10.18%. Applying a published English normalizer to the same verbatim comparison produced 8.29%.

Those are three views of the output, not three attempts from which we picked the most flattering number. The distinction matters throughout this report.

Editorial paper collage of a long audio tape, microphone and research notebook.
Full recordings give the test room to encounter more than a clean opening paragraph. Illustration; the measured results follow below.

We make Podtyper and ran this evaluation ourselves. This is a test of recognizer responses, using the vocabulary extraction behind our podcast transcription workflow. It is not an independent certification or a measurement of every step in the app, including downloads, chunk assembly, speaker labels and summaries.

What went into the test

Rev16 is a public set of podcast recordings and human references maintained in Rev’s speech-datasets repository. We used all 16 available recordings rather than selecting episodes after seeing how the recognizer performed.

Twelve recordings are conversational. The other four are a trailer, a fiction reading, a sermon and a solo advice episode. We retain all four in the full-set score and label them separately; calling the entire collection conversational would overstate the coverage.

Cropped Rev16 repository directory showing separate nonverbatim and verbatim transcript folders, a license file and the podcast catalog.
Actual directory contents at the pinned Rev16 revision. Both human reference styles remain available for inspection. Source; captured 27 October 2026 (UTC).

The files range from about three minutes to just over two hours. That is a useful improvement over a short-clip test, although it still represents a small English-language collection. Sixteen recordings are not sixteen languages, thousands of accents or a representative sample of every podcast format.

We used grok-voice-transcribe-2.0 through xAI’s speech-to-text endpoint. Each request received a complete source file. Diarization was off, language detection was automatic, and we applied no transcript cleanup. The run made 35 requests: two conditions for each recording, plus three title-only requests. There were no failed calls, inference retries or best-of-several selections.

The 35 requests reuse the same recordings. The coverage is 16.15 unique audio hours, not 35 independent episodes or twice that duration.

Why we report more than one score

A verbatim reference keeps features of speech such as fillers and repetitions. A nonverbatim reference edits some of those features for readability. A recognizer can agree more closely with one style without having understood more of the conversation.

Our primary comparison was fixed to the nonverbatim reference before inference. We have kept it primary. The table also reports the verbatim comparison and a supplementary normalized view.

Scoring view, all 16 recordingsWithout contextWith metadata hints
Strict nonverbatim WER — original primary measure11.37%11.43%
Strict verbatim WER10.19%10.18%
English-normalized verbatim WER — supplementary8.30%8.29%

“Strict” here means our original fixed text-scoring rules, not literal character-by-character matching. The supplementary view uses the English normalizer from a pinned revision of Whisper. It accepts additional text equivalences, including some number and contraction forms. We applied it to both the hypothesis and reference.

The normalizer changed the scoring rules. It did not repair the transcript. Our word error rate explainer works through the arithmetic and shows why the reference-word denominator can change too. The pinned normalizer implementation makes those equivalence rules inspectable. Both scoring views remain in the downloadable evidence; the normalized score is not semantic error rate.

The average hides the useful part

Some recordings were much easier than others. Seven had normalized verbatim WER below 5%; eleven were below 8%. A single full-set number cannot tell you which kind of episode will need the most checking.

Normalized verbatim WER for all 16 recordings, with conversational recordings distinguished from four other formats.
Same metadata-hint condition and scoring view for every recording. The dashed line is the word-weighted corpus WER. Gray points identify the four non-conversational recordings; they remain in the aggregate.

We calculate corpus WER by adding the word errors and dividing by the total number of reference words. We do not average sixteen percentages equally. A two-hour episode contributes more reference words than a three-minute trailer.

On the twelve conversational recordings alone, normalized verbatim WER with hints was 8.82%. The corresponding primary strict nonverbatim figure was 12.06%. That subset helps describe the intended use case, but it does not replace the complete-set result.

Names improved, with a tradeoff

An overall word score can look reasonable while a guest’s name is consistently wrong. We therefore checked a vocabulary list frozen before transcription, counting correctly aligned occurrences rather than whether a name appeared correctly once somewhere in the episode.

Against the nonverbatim references, metadata hints increased selected-name recall from 31/57 to 43/57 occurrences. Domain-term recall changed from 101/110 to 103/110.

There was a cost. Selected-name precision fell from 91.18% to 75.44%. The hints recovered more reference occurrences, but the output also contained more checklist-name mentions that did not match the reference alignment. These are reference disagreements; we have not independently listened to and adjudicated each one as an invented name.

That makes “context improved accuracy” too broad a conclusion. It helped one part of the task and hurt another. The paired vocabulary counts below use the same nonverbatim references and fixed checklist in both conditions.

Selected vocabularyReference occurrencesCorrect without hintsCorrect with hints
People's names5731 (54.39%)43 (75.44%)
Domain terms and named topics110101 (91.82%)103 (93.64%)
Brands and platforms257251 (97.67%)252 (98.05%)

These are occurrence counts, not an exhaustive entity benchmark. First-name and full-name checklist entries can overlap. Hints came from metadata available before transcription, rather than spellings repaired after inspecting the output. For the paired recall and precision charts, see our keywords and names experiment.

What the context comparison actually supports

The primary full-set WER changed from 11.37% without context to 11.43% with context, a difference of +0.060 percentage points. A show-cluster bootstrap interval was −0.030 to +0.203 percentage points. Negative values would favor hints.

The interval includes zero. On this collection, the experiment does not establish a consistent overall WER improvement from metadata. It also does not prove that hints are useless: the selected-name recall result measures a different, narrower outcome.

We resampled at the show level because episodes from the same show are not fully independent. The calculation used 10,000 resamples of twelve shows, with seed 20261027. It describes uncertainty within this small collection, not a guarantee about the next episode a customer uploads.

We also tried different recognizer settings

The follow-up experiment added 59 requests. We tested eight configurations, selected one using a frozen rule, and compared it with fresh default requests on other shows in the same corpus.

Disabling the voice-activity gate produced a small apparent WER improvement during development. In confirmation, the improvement was too small and uncertain to pass our rule, while selected-name F1 fell. We kept the existing defaults. On the eleven confirmation recordings from nine shows excluded from settings development, normalized verbatim WER changed from 8.02% to 7.99%. The paired difference was −0.037 percentage points, with a 95% show-cluster interval of −0.089 to +0.043 points. Selected-name F1 fell from 0.873 to 0.793. The candidate did not pass the predefined confirmation gate.

Those recordings were part of the original corpus, so this was confirmation on other shows within that corpus, not a new unseen test set. The follow-up kit retains the configurations, frozen selection rule and all responses. Our full Grok settings experiment walks through the development results and confirmation gates.

What we have not measured

This run does not establish a current ranking against Descript, Sonix, Otter or another app. That requires the same files, reference versions, scoring rules and actual product workflows. An external engine score is useful context, but it is not a result for the app built around that engine.

Our podcast transcription tools guide compares documented workflows separately from these recognizer measurements. Its product recommendations are not a shared accuracy ranking. Our guide to comparing transcription benchmarks explains how to read external results without combining different experiments into one leaderboard.

We also did not measure human correction time, verify every quotation, score overlapping-speaker passages separately or establish speaker-label accuracy. The dataset is public, so we cannot rule out prior exposure during model training. A fresh private test would answer a different and valuable question.

Before publishing a quotation, a name or a consequential number, the practical step remains simple: check the relevant audio. WER counts text disagreement; it does not decide which disagreement matters to your reader.

Reproduce the result

The original benchmark kit contains the frozen plans, scorer and retained evidence. The follow-up settings kit adds the normalizer, settings protocol and confirmation outputs. A compact results file supplies the numbers used for this series’ charts.

For smaller downloads, inspect the frozen audio and reference manifest, request plan, strict occurrence scores and normalization audit. The archived technical report preserves the detailed tables and earlier audiobook write-up. The flagged listening excerpts remain unreviewed; automated flags are not independently verified recognition errors.

The dataset reference revision is c05ab6fd8b4b627d123c922a22a39e993dd37635. The audio mirror is pinned separately, and file hashes are retained. The earlier audiobook kit remains a separate download; its results should not be pooled with the podcast run.

Rev’s references carry their upstream attribution and license information in the kit. Audio rights remain with the original publishers; we link to the source recordings rather than bundling or rehosting the episodes.

This is stronger evidence than the audiobook test because it covers complete podcasts and exposes where the result is uneven. It is still a measured result for one defined collection. That is the claim we can stand behind.

Try Podtyper free — no credit card needed

Paste a supported public podcast link and download PDF or TXT. Your first 30 minutes are free.

Start transcribing