AI Engineering Stack
Caching & Cost Optimization
Strategies for drastically reducing latency and API costs in production.
Interview: High - Every company wants to reduce their OpenAI bill.
Why Cache LLM Calls?
Caching Layers
LLM API calls are expensive and slow. If 1,000 users ask "What is your return policy?", you shouldn't generate the answer 1,000 times.
1. Exact Match Caching
Using Redis or Memcached. You hash the exact prompt and hyperparameters as the key, and store the text response as the value. Extremely fast, but fragile (a single space changes the hash).
2. Semantic Caching
Instead of matching exact strings, semantic caching uses embeddings. You embed the user's prompt, query a vector database (like Redis, Pinecone, or Milvus) for similar previous queries. If a match exceeds a similarity threshold (e.g., 0.95), you return the cached response.
Cost Optimization Strategies
- Model Routing: Send simple tasks (summarization, extraction) to cheaper/faster models (Claude 3 Haiku, Llama 3 8B). Send complex tasks (reasoning, coding) to GPT-4o or Claude 3.5 Sonnet.
- Prompt Minimization: Strip unnecessary whitespaces, comments, and boilerplate from the prompt to reduce token count.
- Batch Processing: For non-real-time tasks (e.g., processing 10,000 resumes overnight), use the OpenAI/Anthropic Batch APIs for a 50% discount.
Use Cases
E-commerce FAQ bots fielding redundant questions
Reducing latency from 5000ms to 50ms for repeat code generation queries
Dynamic model routing based on user input intent
Common Mistakes
Using Semantic Caching for tasks that require real-time dynamic data (e.g. stock prices)
Setting the semantic similarity threshold too low, returning irrelevant cached answers
Failing to implement cache invalidation when underlying system knowledge changes