Writing / 25 July 2026

Three speech engines, and why I shipped the least accurate one

Written 25 July 2026, when the pipeline ran a 7B model. It now runs Qwen 3.5-4B; the speech results below still hold.

Whisper transcribed my test set at 1.1% word error rate. Apple's built-in engine got 3.1%. I took Whisper out of the app anyway, and the reasons came down to download size and heat.

The constraint

I'm building a pipeline that transcribes a clinical conversation and has a language model evaluate it, all on an iPhone. Nothing can go to a server. That rules out every hosted speech API and leaves a trade between accuracy, download size and heat. A 480 MB first-run download matters when the person installing the app has a few minutes between patients.

How I measured

I wrote synthetic doctor–patient conversations with known text, synthesized the audio with macOS say, ran each engine and computed word error rate (WER): the share of words inserted, deleted or wrong. Both sides are lowercased and stripped of punctuation before scoring, so formatting doesn't count for or against anyone.

The candidates were Whisper small.en (OpenAI's model on MLX, about 480 MB), Parakeet TDT (NVIDIA's model, which came bundled with the diarization library I was using) and Apple SpeechAnalyzer, which is built into iOS and macOS 26 and needs no download.

Results

I never measured Parakeet. Its MLX weights alone were a 75-minute download on the test machine, and by the time that mattered I'd already decided.

The Apple rerun used 30 conversations (5,555 reference words, 28.5 minutes of audio) and landed within 0.1 points of the first run. Split by length:

overall             3.1%  (n=30)
short  (≤10 turns)  4.0%
medium (11–24)      3.0%
long   (25+)        3.1%

Accuracy doesn't fall off as conversations get longer, which was the failure I was most worried about. Median 3.0%, standard deviation 1.6%, worst case 7.8%.

The number that mattered most wasn't accuracy. Apple transcribed the 28.5 minutes of audio in 43 seconds, about 40× realtime, at roughly 6% CPU, because it runs on the Neural Engine instead of the GPU.

Why the less accurate engine won

In a 300-word conversation, 1.1% WER is about 3 wrong words and 3.1% is about 9. The transcript only feeds a language model that reads for meaning, and its verdicts didn't change between engines.

On Apple's side of the ledger: no 480 MB download, which also meant no download screen or resume logic to build. It ran at 40× realtime on the Neural Engine, leaving the GPU to the language model, which was already making the phone hot. Removing Whisper also deleted a dependency and shrank the app.

WER also weights every word the same. "The" counts as much as "no chest pain", and in a clinical transcript a dropped negation matters far more than a dropped filler word. After seeing that, I stopped reading a 2-point WER gap as meaningful.

Diarization model vs a language model

The evaluation needs to know which lines belong to the clinician. The usual answer is a diarization model that segments audio by voice. I tested skipping it: give the language model a flat transcript and ask it to split the speakers itself. Over 15 cases it separated the two voices correctly 73.6% of the time and got the roles right 68.1% of the time.

That was too low, but the failure was specific. The model was decent at judging who a line sounds like and bad at finding where one speaker stops. Apple's streaming transcriber already breaks its output at pauses, which is usually where speakers change, so I kept those boundaries and asked the model only to label each utterance. The diarization model came out of the app. With Qwen 3.5-4B that labeling now reaches 98.3% on real recordings, against 85.0% for the 7B.

Limits

Every number above is clean synthetic speech, so real audio with accents and room noise will do worse. Overlapping speech is untested, because each line was synthesized separately. Medical vocabulary is untested too, and an engine that handles ordinary English can still mangle "levothyroxine".

I shipped on a weaker kind of evidence: weeks of real recordings on real phones where Apple's transcript was never the reason an evaluation came out wrong.

Where the error lives

All three engines land between 1% and 3% on clean speech. The errors that hurt my output came from the language model's judgment, so I went in expecting the benchmark to pick a speech engine and came out spending the next month on the model.