Speech recognition, or speech to text, turns sound into written words in three broad steps. First the audio is converted into a picture of its frequencies over time. Then a neural network reads that picture and builds an internal representation of what was said. Finally a decoder writes out text, one small piece at a time, choosing the most likely next piece given both the sound and the words so far. That last step is why context helps and why unusual names and numbers are where most mistakes happen.
This guide uses OpenAI's Whisper as the worked example, because its paper describes each step in detail.
Step 1: How does sound become something a model can read?
A microphone records air pressure as a stream of numbers. Speech models do not read that stream directly. They convert it into a spectrogram: a grid showing how much energy is in each frequency band, moment by moment.
The Whisper paper gives the exact recipe: "All audio is re-sampled to 16,000 Hz, and an 80-channel log-magnitude Mel spectrogram representation is computed on 25-millisecond windows with a stride of 10 milliseconds." In plain terms: the audio is measured 16,000 times a second, cut into overlapping 25 ms slices every 10 ms, and each slice is summarised as 80 numbers spaced the way human hearing spaces pitch (the Mel scale). The large-v3 model card says the newest large model uses 128 frequency bins instead of 80.
Whisper also works in fixed chunks. The paper says it was trained on "30-second segments," and longer recordings are transcribed by sliding that 30-second window along. It notes that "inaccurate transcription in one window may negatively impact transcription in the subsequent windows."
Step 2: What does the encoder do?
The encoder is a neural network that reads the whole spectrogram and turns it into a set of internal features: patterns that capture sounds, syllables and their context. Whisper's paper describes "an encoder-decoder Transformer," where the encoder first passes the spectrogram through "two convolution layers" and then a stack of Transformer blocks.
Older systems split this job into separate parts: an acoustic model for sounds, a pronunciation dictionary, and a language model for likely word sequences. OpenAI's Whisper README describes the newer approach as "allowing a single model to replace many stages of a traditional speech-processing pipeline."
Not every modern model is built the same way. NVIDIA's Parakeet TDT models, for example, pair a Conformer encoder with a transducer-style decoder, according to the model details on the Open ASR Leaderboard. The broad idea, sound in and text out through a learned network, is the same.
Step 3: How does the decoder write the text?
The decoder produces text as tokens, which are words or pieces of words. Whisper uses "the same byte-level BPE text tokenizer used in GPT2," per the paper, so a rare surname may be built from several fragments.
Before writing any words, Whisper's decoder is given a short set of instruction tokens: which language is being spoken (the paper lists 99), whether to "transcribe" or "translate", and whether to add timestamps. If there is no speech, it is trained to predict a special "nospeech" token instead.
Then it writes, one token at a time. At each step it picks the most likely next token given two things: the encoder's view of the audio, and the text it has already written. The paper calls the decoder "an audio-conditional language model," and says it was trained to use the previous transcript text "to resolve ambiguous audio."
Why do context and vocabulary matter so much?
Because the decoder is partly a language model, it leans towards word sequences it has seen often. That helps with everyday speech: if the audio between "for" and "example" is unclear, "for example" is a good bet. It hurts with words it has rarely seen.
That is why most tools let you add context:
- OpenAI's Whisper code has an
initial_promptoption which, per its transcribe code, can supply "custom vocabularies or proper nouns to make it more likely to predict those word correctly." - Apple's Speech framework lets developers "bias recognition towards certain words" and "supply custom vocabulary," according to its DictationTranscriber documentation.
- Dictation apps often expose this as a personal dictionary of names and acronyms.
Why do names and numbers go wrong?
| What goes wrong | Why |
|---|---|
| A surname becomes a common word | The decoder prefers frequent words that sound similar |
| "Fifteen" heard as "fifty" | Short, similar-sounding words; one syllable decides the meaning |
| "14" vs "fourteen" vs "14th" | Many correct written forms; the model learned whichever styles its training transcripts used |
| A "not" disappears | One short, often unstressed word that is easy to miss, yet it reverses the meaning |
| Words appear that you never said | Hallucination, often over silence or noise |
On the last point, the large-v3 card is direct: predictions "may include texts that are not actually spoken in the audio input (i.e. hallucination)." It explains that the models "combine trying to predict the next word in audio with trying to transcribe the audio itself." It also warns that the architecture is "prone to generating repetitive texts." Why words appear after silence covers what that looks like in a dictation app.
Accents, microphones and rooms matter too. The same card says the models "exhibit disparate performance on different accents and dialects."
What does cleanup or formatting add afterwards?
Whisper already writes punctuation and capitals. The paper says OpenAI trained it "to predict the raw text of transcripts without any significant standardization," which "removes the need for a separate inverse text normalization step." Inverse text normalisation is the old step that turned "fourteen dollars" into "$14" and added commas.
Dictation apps usually add a further pass on the text after recognition:
- Cleanup: fixing spacing, capitals at sentence starts, number formats and stray filler words.
- Dictionary replacement: swapping known misrecognitions for the names you taught it.
- Optional AI rewrite: sending the text to a language model to tidy grammar or change tone. This may run locally or in the cloud, and it can change meaning, so read what it returns.
This step works on text, not sound. It cannot recover a word the recogniser never caught.
Where does SuprFlow fit?
SuprFlow 1.0.0 follows this pipeline on the Mac. Recognition uses a WhisperKit (Whisper) model, with Small and Medium options in multilingual and English-only variants. Its Default formatting is local cleanup of punctuation, capitalization and numbers, and it has a personal dictionary for names and acronyms. An optional Cloud API mode sends the transcript text, not the audio, to a provider you configure. Teach SuprFlow names and acronyms shows the dictionary step.
Questions people ask
What is speech to text?
Software that converts spoken words into written text. It powers dictation, captions and transcription. The recognition step can run on your own device or on a provider's servers.
Why does speech recognition get my name wrong?
Names are rare in training data, so the decoder tends to choose a more common word that sounds similar. Adding the name to a dictionary or prompt, where the tool allows it, usually helps more than changing models.
Does Whisper add punctuation by itself?
Yes. It was trained on raw transcripts that include punctuation and capitals, so it writes them as part of recognition. Apps may still tidy the result afterwards.
Can speech recognition work without the internet?
Yes, if the model runs on your device. On-device speech recognition explained covers what that means and how to check.
Related guides
- On-device speech recognition explained
- Speech to text accuracy and word error rate
- Whisper model sizes compared
- Local vs cloud formatting
- Improve Mac dictation accuracy
- Parakeet vs Whisper