Prompt Caching & Speculative Decoding

Stop Paying to Process the Same System Prompt 10,000 Times

Prompt Caching Flow

If your system prompt is 2,000 tokens and you have 100,000 requests/day, you are spending 6/day (2,160/year) just on re-processing the same static context. Prompt Prefix Caching lets Anthropic and Google compute and cache your static context on their infrastructure, so you only pay for input tokens once and cache reads at 90% discount.

How Prefix Caching Works

The model provider hashes the prefix (your system prompt + any static context). On subsequent requests with the same prefix, the KV cache for those tokens is served from their GPU memory instead of recomputed. The constraint is that the prefix must be identical byte-for-byte. Even a single character change invalidates the cache. This is why dynamic content (user names, timestamps) must always be appended after the cached prefix, never injected inside it.

Speculative Decoding

Standard decoding is sequential and slow: 1 token per compute step. Speculative Decoding uses a small, fast "draft" model (e.g. Llama 3 8B) to generate 4-8 tokens in parallel, then uses the large "oracle" model (e.g. Llama 3 70B) to verify them all in a single forward pass. If the draft tokens match what the large model would have generated, all are accepted simultaneously. This achieves 2-3x throughput improvement for free with no quality loss.