LLM Inference Engineering
Quantization: GGUF, AWQ & EXL2
Model compression formats, precision trade-offs, and running 70B models on consumer hardware.
Interview: High - Essential for cost-efficient model deployment.
You Don't Need a $40,000 GPU Cluster
Quantization Precision Levels
A Llama 3 70B model in full BF16 precision requires 140GB of VRAM. No single consumer GPU has that. Quantization compresses model weights from 16-bit floating point to 4-bit or 8-bit integers, reducing memory requirements by 4-8x while retaining 95%+ of model quality.
Quantization Format Comparison
- GGUF (llama.cpp): CPU-first inference. Runs on a MacBook M3 or a Linux server without a GPU. Slow (5-10 tokens/sec) but free. Best for developers who need local prototyping.
- AWQ (Activation-aware Weight Quantization): GPU inference, 4-bit, minimal quality loss. Measures activation magnitudes first to protect salient weights during compression. Preferred for vLLM deployment.
- EXL2 (ExLlamaV2): Mixed-precision quantization assigning different bit widths per-layer based on perplexity sensitivity. Achieves the best quality-at-size trade-off across all formats. Used when quality is critical.
The Quantization-Quality Trade-off
Q4_K_M (4-bit) reduces 70B model from 140GB → 40GB. Quality loss is typically <1% on benchmarks. Q2_K (2-bit) reduces to 20GB but loses 5-8% quality. There is a hard floor: below Q4, you begin seeing significant instruction-following degradation on complex reasoning tasks.
Use Cases
Local development with Llama 70B on a Mac Studio M2 Ultra using GGUF
Production serving with 4x quality-per-dollar improvement using AWQ on A100s
Offline inference on edge devices or air-gapped environments
Common Mistakes
Using Q2 or Q3 quantization for instruction-following tasks — the model loses coherent reasoning ability below Q4
Mixing quantization formats (AWQ model with GPTQ-expecting server) — always verify the quantization format matches your inference server
Not benchmarking quantized model quality on your specific task before deploying — synthetic benchmarks don't always match domain performance