Whisper Architecture: From Audio Waveform to Text

The goal: understand Whisper from the highest level down to the internal transformer operations. That includes the encoder, the decoder, the self-KV cache, the cross-KV cache, the decoding loop, and why these components matter for performance.

I use simple English first, then build up to the implementation details. The diagrams are important, because Whisper is easier to understand as a sequence of data transformations than as a collection of neural-network terms. If you prefer to watch, the video above walks through the same architecture.

Executive summary

Whisper is an automatic speech recognition (ASR) model. It converts spoken audio into text.

The most important idea is this: Whisper has two main neural-network parts.

• Encoder: listens to the audio and converts it into a representation of what was heard.
• Decoder: uses that representation to generate text, one token at a time.

The decoder uses two kinds of attention:

These caches solve different problems. Self-KV grows as text is generated. Cross-KV is reusable for the same encoder output and a compatible decoding setup.

That distinction is central to understanding Whisper’s performance, and to my work on running Whisper inside a trusted execution environment (TEE).

The complete architecture

What happens to the data?

Each stage changes the form of the data.

Audio preprocessing: how sound becomes numbers

A computer does not hear words directly. It receives a waveform: a long sequence of numbers that represents changes in air pressure.

Step 1: The waveform

Imagine someone says: “The server is running.” The microphone records the sound as a changing signal.

In standard Whisper preprocessing, audio is resampled to 16 kHz. This means 16,000 samples represent each second of audio. For 30 seconds:

That gives 480,000 waveform samples for a mono recording.

Step 2: The log-Mel spectrogram

Whisper does not send all those raw samples directly into the transformer encoder. It converts the audio into a log-Mel spectrogram. Think of it as a heat map:

• Horizontal axis: time.
• Vertical axis: frequency bands.
• Brightness or colour intensity: strength of the sound in a frequency band.

The standard Whisper configuration uses 80 Mel-frequency channels, with a 25 ms analysis window and a 10 ms stride. A 30-second input produces 3,000 time frames before the encoder’s convolutional downsampling.

Why use a spectrogram? The waveform tells us how the signal changes. The spectrogram makes frequency patterns over time easier for the neural network to process.

Step 3: Convolutional downsampling

Whisper’s encoder starts with two one-dimensional convolutional layers.

The encoder therefore processes 1,500 positions for a full 30-second input, rather than 3,000 spectrogram frames. This is a major reduction in sequence length.

The audio encoder: understanding the sound

The encoder’s job is to convert the spectrogram into useful representations of the audio. It does not need to decide the exact text token at every position. Instead, it creates representations that the decoder can use to generate the transcription.

What does encoder self-attention do?

Suppose the speaker says: “I deposited money into the bank.” A local sound may be unclear. The surrounding audio can help the model interpret it. Self-attention lets an audio position use information from other positions in the same audio sequence. For example:

• A sound at one position can use information from nearby sounds.
• It can also use information from a later position.
• The encoder can combine information from different parts of the audio.

Unlike the decoder’s masked self-attention, the encoder’s self-attention can normally examine both earlier and later audio positions.

What is the encoder output?

For a full 30-second input in Whisper Tiny.en:

This represents 1,500 audio positions, each with 384 numerical features. The actual tensor also includes a batch dimension.

The encoder processes the audio once for a given input. Its output then becomes the memory that the text decoder reads through cross-attention.

The decoder: generating the transcript

The decoder is an autoregressive transformer. Autoregressive means that it generates an output sequence step by step. It predicts the next token from the tokens already generated and the encoder’s audio representation. Consider the example “The server is running.” A simplified generation sequence might look like this:

This is a simplified illustration. Actual Whisper tokens can represent subword pieces, punctuation, timestamps, or task markers. A single word may need more than one token.

Inside each decoder block

Every standard Whisper decoder block contains three main operations:

1. Masked self-attention: uses the generated text prefix and Self-KV.
2. Cross-attention: uses the encoder output and Cross-KV.
3. Feed-forward network: processes the combined representation.

Each operation also uses residual connections and layer normalization. This pattern repeats across the decoder layers. Each layer has its own attention projections and its own K/V tensors.

The decoder must combine two types of evidence:

• Text context: what has already been generated?
• Audio context: what does the original recording support?

Self-attention supplies the first. Cross-attention supplies the second.

Attention, Query, Key and Value: the foundation of KV caching

Before explaining the caches, we need to understand what attention does. Imagine a library.

• Query (Q): the question you ask.
• Key (K): the label used to identify relevant information.
• Value (V): the actual information you retrieve.

Attention compares a query with keys. It uses the comparison scores to decide how much information to take from each value.

The mathematical operation

For one attention head:

• X_q: input used to generate queries.
• X_k: input used to generate keys and values.
• W_Q, W_K, W_V: learned projection matrices.
• d_k: dimension of each attention head.

The same basic attention operation is used for both self-attention and cross-attention. The difference is the source of X_q and X_k.

What does multi-head attention mean?

Instead of performing just one attention calculation, Whisper divides the hidden representation into multiple attention heads. Each head can learn different relationships. For example, one head might focus on nearby positions, while another might learn longer-range relationships. These are illustrative possibilities, not fixed roles. The outputs of the heads are combined and passed through another learned projection.

Self-KV cache: remembering the generated text

The problem without a cache

Suppose Whisper has generated the tokens The → server → is → running. When it generates the next token, a basic implementation could process the entire prefix again: process The, process server, process is, process running, and calculate all the attention projections again. This repeats work that was already completed, and the repeated work grows as the transcript gets longer.

The solution: Self-KV

For each decoder layer, Whisper stores the keys and values calculated for the text tokens it has already processed. When a new token arrives, Whisper calculates its new K/V entries and appends them to the cache.

What is actually stored?

For each decoder layer:

• The key projection of each processed text position.
• The value projection of each processed text position.

The cache does not normally store the final predicted word as its K/V entry. It stores numerical representations calculated from the decoder’s hidden states. It also does not need to cache the query in the standard K/V caching design, because the query is calculated for the current decoding step.

What happens at the next token?

Suppose the cache already contains K/V entries for The server is. The decoder receives the next input token, calculates its query, key and value, then uses:

The semicolon represents concatenation along the sequence dimension. The new query can attend to the earlier keys and the current key. It then retrieves information from the corresponding values.

The cache saves the need to recalculate the old tokens’ K/V projections. It does not remove the attention calculation against the cached keys and values.

Why is decoder self-attention masked?

During generation, Whisper must not use future text tokens that have not been generated yet. For example, when it predicts is, it cannot use the future token running. The causal mask prevents attention to future positions.

During incremental inference, Whisper usually processes only the newly available decoder token after the initial prompt. The cached prefix supplies the historical K/V entries. The OpenAI implementation uses hooks to append Self-KV entries and uses the cached K/V values on later decoding steps.

Cross-KV cache: remembering the encoded audio

Cross-KV is different. It stores information derived from the audio encoder output, not the generated text.

Why does the decoder need cross-attention?

Imagine Whisper has generated “The server” and needs to predict what comes next. Self-attention tells it what the previous text says. But previous text alone is not enough. The decoder also needs to consult the recording to decide whether the speaker said is, was, or something else. Cross-attention provides this connection to the audio.

Where do Cross-K and Cross-V come from?

The encoder produces the audio representation H. Each decoder layer has its own cross-attention key and value projections:

The exact matrices differ by decoder layer. These projections are calculated for the encoder output and cached. On later decoding steps, the decoder reuses the same Cross-KV tensors while it uses new queries derived from the evolving text.

Why can Cross-KV be reused?

The audio representation is fixed for a particular encoder result. The decoder’s generated text changes, but the audio representation does not change because another text token is generated. So for the same audio encoder output and a compatible model setup, the decoder can reuse the same Cross-KV tensors instead of projecting the entire audio representation again on every token step.

Pay attention to the exact condition: the reuse is valid only when the cached projections correspond to the encoder output and model weights being used. If the audio is re-encoded into a different representation, the previous Cross-KV is not automatically valid.

What changes during generation?

An important implementation detail

The two caches may share the same dictionary in the reference implementation, but they are not one interchangeable cache. The OpenAI code distinguishes them through the attention modules and the shape and reuse logic:

• Self-attention K/V entries are extended as the text prefix grows.
• Cross-attention K/V entries are computed from the encoder output and reused.

This detail matters when you read the actual source code. A cache dictionary is just the container. The meaning of a cache entry depends on which layer and projection produced it.

Self-KV versus Cross-KV: one complete example

Suppose the audio says “Transfer the money tomorrow.” Whisper encodes the audio once, then starts decoding.

The key distinction is not that one cache is important and the other is not. Both reduce repeated computation, but they preserve different information.

Inside one decoder layer: where the caches are used

Here is how the two caches connect to the actual computation inside a decoder block.

The model uses residual connections, so each sublayer adds its transformed result to its input. Layer normalization keeps the numerical values well behaved. The feed-forward network uses two learned linear transformations with a GELU activation between them.

A subtle but important point about queries

At a given decoding step:

• The current token produces a new self-attention query.
• The current token also produces a cross-attention query.
• The Self-KV cache supplies the accumulated text keys and values.
• The Cross-KV cache supplies the keys and values derived from the audio.

The queries are not shared between the two attention operations. Each attention module has its own learned projection matrices.

Whisper model sizes and architecture dimensions

Whisper uses the same general encoder-decoder design across several model sizes. The number of layers, attention heads and hidden dimensions changes. The table lists the standard OpenAI model configurations. Values are per model, not per batch.

The standard OpenAI tiny.en configuration uses the Tiny-sized architecture. Its English-only vocabulary differs from the multilingual vocabulary, but the broad architecture is the same.

What do these dimensions mean in practice?

For tiny.en:

• Encoder hidden dimension: 384.
• Encoder attention heads: 6.
• Decoder hidden dimension: 384.
• Decoder attention heads: 6.
• Encoder layers: 4.
• Decoder layers: 4.
• Audio context length: 1,500 positions for a standard 30-second input.
• Text context limit: 448 positions.

Each attention head has a dimension of:

A single Self-K tensor at one decoder layer contains a sequence dimension, six heads, and 64 features per head, conceptually:

Similarly, Cross-KV has an audio sequence dimension:

Here, B is the batch size and T_text is the number of cached text positions. The reference implementation stores K/V in a layout that is first represented as batch × sequence × hidden dimension, then reshapes it into attention heads during computation.

These shapes explain why the two caches behave differently. Self-KV grows along the text dimension, while Cross-KV stays fixed along the audio dimension for a given encoded audio input.

Why Whisper has task tokens

Whisper is not limited to transcribing English speech into plain text. It supports several related tasks through a shared model. Special tokens tell the decoder what kind of output is expected.

For example, a multilingual transcription prompt can conceptually contain start of transcript → language → transcribe → text. A translation prompt instead signals the translation task. Whisper then generates English text from the speech, even if the source audio is in another language. These are special tokens in the model vocabulary, not ordinary words. The exact prompt sequence depends on the task and decoding options.

Why this matters for KV caching

The initial prompt is also processed by the decoder, and its token representations contribute entries to Self-KV. So the cache is not limited to the words in the transcript. It can also contain entries for the task prompt and other preceding tokens.

How the complete decoding loop works

The model does not generate the entire transcript in one decoder step. The usual inference process is iterative.

The loop, in simple terms

1. Convert the audio into a log-Mel spectrogram.
2. Run the encoder to produce audio features.
3. Prepare the initial task prompt.
4. Run the decoder and initialise the caches.
5. Predict the next token.
6. Select a token with the configured decoding strategy.
7. If generation is not complete, feed the new token into the next decoder step.
8. Append new Self-KV entries and reuse Cross-KV.
9. Continue until the task ends or a decoding limit is reached.
10. Convert the generated tokens into the final output.

Whisper can use greedy decoding or beam search, depending on the configuration. With beam search, it may keep multiple candidate sequences. The implementation must keep the corresponding Self-KV state aligned with the selected candidate sequences.

KV cache memory: how much space does it need?

Caching reduces repeated computation, but it uses memory. This is an important trade-off.

A simple formula

For a standard multi-head attention layer, the combined size of its K and V tensors is approximately:

• B: batch size.
• L: number of layers.
• T: sequence length.
• d: hidden dimension.
• s: bytes per stored element.
• The factor 2 represents Key and Value.

This is a simplified formula for one cache type. It assumes the same hidden dimension and sequence length across layers.

Example: Whisper Tiny.en

Assume batch size B = 1, four decoder layers, hidden dimension d = 384, FP16 cache elements (s = 2 bytes), 100 text positions in Self-KV and 1,500 audio positions in Cross-KV.

These estimates exclude the encoder output itself, model weights, temporary attention tensors, allocator overhead and other runtime memory.

Why KV caching improves performance

Without caching, the decoder may repeatedly project the same historical text and audio representations into K/V tensors. With caching:

• Previously processed text K/V can be reused.
• Audio K/V projections can be reused for the same encoder output.
• The decoder can process only the newest text token in incremental generation.

However, the decoder still performs attention calculations against cached entries. The cache reduces repeated projection work, not every operation in attention.

There is another trade-off: caching uses memory bandwidth and storage. If the cache is too large, moving and reading the cached tensors can become expensive. This matters especially when inference runs across separate processes, devices or protected environments.

Whisper in my SGX/TEE research

My work on native-sgx-whisper makes one distinction particularly important: reusing a cache is not the same as reusing a valid cache. Here is the relevant data path.

This is a conceptual architecture. It does not imply that every component runs inside the same enclave, or that cache contents are automatically protected during every transfer. That depends on the actual implementation and trust boundaries.

Cross-KV reuse: the critical constraint

Consider two different audio chunks:

• Chunk A: seconds 0 to 30.
• Chunk B: seconds 20 to 50.

Even though they overlap for ten seconds, their encoder outputs are not automatically interchangeable. The encoder can use context across positions, so a different chunk boundary can change the representation of even overlapping audio. Therefore, Cross-KV for Chunk A cannot be assumed valid for Chunk B.

Self-KV reuse: a different constraint

Self-KV represents the decoder’s previous hidden states. Suppose the decoder generated a prefix for Chunk A. It is tempting to reuse that Self-KV for Chunk B, because some words may be the same. That is not automatically correct. The hidden states depend on the decoder context, including the token prefix and positions. If the prefix, the token positions or the decoding state changes, old Self-KV may no longer represent the required state. The same words do not guarantee the same cached representations.

Why this matters for my performance experiments

For a Whisper Tiny.en pipeline, separate these potential costs:

A cache reuse optimization should also be tested for transcript equivalence or quality. A faster output that changes the transcription is not an equivalent optimization.

My earlier experiments found that the skip_encode/KV reuse path caused transcript divergence. The right conclusion is that this tested reuse path was not correct for the tested conditions, not that all Cross-KV caching is unsafe. The reference Whisper implementation itself uses Cross-KV caching for unchanged audio features. The issue is whether the particular state being reused is valid.

Training versus inference

Whisper’s architecture is easier to understand if we separate training from inference.

Training

During training, the model is given audio and the corresponding reference text. The decoder learns to predict the next token from the preceding reference tokens and the encoder output. For example:

• Audio: a recording of someone speaking.
• Reference: The server is running.
• Decoder input: a shifted version of the reference token sequence.
• Training target: the next token at each position.

The model compares its predicted token probabilities with the reference targets, and the training loss is used to update the model weights. The full target sequence can be processed in parallel with causal masking.

Inference

During inference, the correct transcript is not available in advance. The decoder must select its own next token, then use that token as part of the next input. It predicts one step at a time, updates Self-KV and reuses Cross-KV for the same encoded audio. This is why incremental decoding and KV caching are particularly useful during inference.

Technical definition tree

Use this tree as a reference when you read Whisper source code or papers.

Whisper ASR
├── Audio preprocessing
│   ├── Waveform
│   ├── Log-Mel spectrogram
│   └── Convolutional downsampling
├── Audio encoder
│   ├── Positional embeddings
│   ├── Multi-head self-attention
│   └── Audio representation H
├── Text decoder
│   ├── Token embeddings
│   ├── Masked self-attention
│   │   └── Self-KV cache
│   ├── Cross-attention
│   │   └── Cross-KV cache
│   ├── Feed-forward network
│   └── Vocabulary logits
└── Decoding and postprocessing
    ├── Greedy or beam search
    ├── Special-token handling
    └── Text and optional timestamps

Key technical terms

Final mental model

Remember these three pictures.

The single most important distinction:

• Self-KV: derived from decoder text states. Its sequence length grows as the decoder processes more tokens.
• Cross-KV: derived from encoder audio states. Its sequence length stays fixed for the same encoder output.
• Neither cache replaces attention. Both let attention reuse previously computed K/V tensors.

The next step is to trace these operations in OpenAI’s implementation:

• model.py: encoder, decoder, attention projections and cache hooks.
• decoding.py: incremental decoding, token selection and cache lifecycle.

Confidence and sources

I have high confidence in the architecture and in the distinction between Self-KV and Cross-KV. These descriptions are consistent with the original Whisper paper and the OpenAI reference implementation. The cache memory figures are illustrative calculations, not measured runtime values. Whether cache reuse is valid across protected execution components depends on the actual encoder input, decoder state, model configuration and implementation.

• Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision” (the Whisper paper).
• openai/whisper reference implementation.
• My video walkthrough of the Whisper architecture.