How Transformer Models Work & KV Cache

Beyond Basic Attention

Transformer Architecture

While the original 2017 Transformer paper laid the groundwork, modern production models use heavily optimized variants. To engineer performant systems, you must understand their architectural bottlenecks.

1. The KV Cache Bottleneck

During auto-regressive decoding, the model generates one token at a time. Recomputing the Key (K) and Value (V) matrices for all previous tokens at every step is computationally catastrophic. The KV Cache stores these calculated vectors in GPU VRAM.

The Engineering Impact: The size of the KV cache grows linearly with context length. A 128k context window can consume over 30GB of VRAM just for the cache (not the weights). This is why providers charge heavily for context window and why techniques like Grouped-Query Attention (GQA) were invented—to compress the KV cache by sharing keys and values across multiple query heads.

2. Rotary Position Embeddings (RoPE)

The original sinusoidal positional encodings failed to extrapolate to longer sequence lengths. RoPE encodes absolute position using a rotation matrix and naturally captures relative dependency between tokens mathematically. This allows models to gracefully handle 100k+ contexts.

Core Modern Optimizations

• RMSNorm (Root Mean Square Normalization): Replaced LayerNorm because it drops the mean-centering step, saving compute by 10-20% without losing quality. Pre-normalization is now standard.
• SwiGLU Activations: Replaced ReLU in the Feed-Forward Network. It acts as a continuous gating mechanism, yielding highly improved empirical scaling performance.
• FlashAttention-2: Mathematically restructures attention to stay exclusively inside the GPU's ultra-fast SRAM, bypassing slow HBM reads. This fundamentally enables 1M+ context windows.