LLM Inference Engineering
vLLM & PagedAttention
Self-hosting open-source LLMs, continuous batching, and CUDA memory management.
Interview: High - Required for any team building cost-sensitive AI products.
Why $0.01/1K Tokens Kills Your Margin
vLLM PagedAttention Architecture
At scale, OpenAI API costs become your biggest infrastructure expense. A team processing 10M tokens/day at GPT-4o pricing spends $150,000/month. At that point, self-hosting an open-source model (like Llama 3 70B) on a cluster of H100 GPUs via vLLM is almost always cheaper and gives you full control over the model, data, and throughput.
PagedAttention: CUDA KV Cache Management
The fundamental problem in GPU inference is KV cache memory fragmentation. With standard PyTorch, when 8 concurrent requests each have different sequence lengths, you must pre-allocate the maximum possible KV cache size for all of them, wasting up to 60-80% of VRAM.
PagedAttention (vLLM's core innovation) borrows the OS virtual memory paging concept. KV cache vectors are stored in fixed-size pages (blocks) which are dynamically allocated on demand. Multiple sequences can share non-contiguous physical KV pages, eliminating fragmentation and enabling 3-4x higher throughput on the same hardware.
Continuous Batching vs. Static Batching
Static batching waits until a full batch of requests arrives before running an inference step. If Request A finishes in 10 tokens and Request B needs 500, the GPU sits idle waiting for B. Continuous batching (also called iteration-level scheduling) inserts new requests mid-batch the moment a slot frees up. This is why vLLM's throughput can be 24x higher than naive Hugging Face inference.
Use Cases
Self-hosting Llama 3, Mistral, or Qwen for data privacy compliance (no data leaving your network)
Reducing inference costs by 10-20x vs. OpenAI/Anthropic at high volume
Building on-premise AI for regulated industries (Healthcare, Finance, Government)
Common Mistakes
Running Llama 70B in full bfloat16 without quantization — requires 140GB VRAM (2x H100). Use AWQ or GPTQ quantization.
Using Hugging Face pipeline() for production inference — it uses static batching and will be 10x slower than vLLM
Ignoring tensor_parallel_size — if you have 4 GPUs, use all 4 for lower latency per request