A KV cache (key-value cache) is an inference-time memory optimization used in transformer-based language models where the attention “keys” and “values” computed for previous tokens are stored and reused for subsequent decoding steps. By caching these intermediate tensors, the model avoids recomputing attention projections for the entire prompt at every new token, significantly reducing latency and compute cost during autoregressive generation.
What is KV Cache (Key-Value Cache)?
Transformers generate text one token at a time. For each new token, self-attention needs access to representations of all prior tokens. Naively, the model would recompute the key and value projections for every token in the context on every step, causing generation cost to grow with sequence length.
KV caching changes this. During the first forward pass over the prompt, the model computes key and value tensors for each layer and stores them. When generating the next token, the model only computes the new token’s query/key/value, appends the new key/value to the cache, and performs attention against the cached history. This makes incremental decoding much faster.
KV cache size grows roughly with: number of layers × sequence length × hidden size (and number of attention heads), multiplied by the numeric precision used (FP16, BF16, INT8, etc.). Because of this, KV cache is often the primary memory bottleneck for serving LLMs, especially with long contexts and high concurrency. Techniques like paged attention, KV cache quantization, and prefix caching are used to make KV cache more efficient in production.
Why it matters in LLM serving
KV caching is critical for real-time chat and agent experiences. It reduces time-to-first-token after the prompt is processed and improves tokens-per-second during decoding. However, it also forces infrastructure tradeoffs: GPU memory must hold the cache for each active request. As you increase context length, batch size, or concurrent users, KV cache memory pressure can reduce throughput or trigger out-of-memory failures.
Operationally, teams monitor KV cache utilization, choose precisions, and adopt memory managers (e.g., paged/persistent caching) to balance cost and performance. For long-context models, KV cache efficiency can matter as much as raw model FLOPs.
Examples
- Chatbot serving: Store KV tensors after processing the conversation history so each assistant token is generated with minimal recomputation.
- Prefix caching: Reuse the same cached prefix (system prompt + tools) across many requests.
- Long-context optimization: Quantize KV cache to INT8 to fit more concurrent sessions on one GPU.
FAQs
Is KV cache used during training? Typically no. It is primarily an inference optimization for autoregressive decoding.
Why does KV cache consume so much memory? Because it stores per-layer tensors for every token in the context window; long contexts multiply memory usage.
Does KV caching improve time-to-first-token (TTFT)? It mainly improves decoding speed after the prompt is processed. TTFT is driven by prompt processing, though prefix caching can reduce it.
Can KV cache be shared across users? Only when the prefix is identical and permissions allow it. Otherwise caches are isolated per request/session.
