Design a Recommendation System

This case study simulates a realistic FAANG system design interview for architecting a real-time personalized recommendation service similar to Amazon's related items or YouTube's home feed. It walks through the two-stage ML funnel, vector indexing, feature store caching, and A/B test validation.

1. Present the Interview Question

2. Clarifying Questions

The candidate clarifies user scale, context parameters, and model latencies:

• Candidate: What is our target query throughput (QPS) for feed requests?
  Interviewer: We expect an average read load of 600 QPS for recommendation queries, but we must process up to 120,000 user interaction logs (clicks, purchases) per second to update preferences.
• Candidate: What is the strict latency budget for retrieving recommendations?
  Interviewer: Recommendations must render in under 100ms.
• Candidate: How quickly must new user clicks affect their live recommendations?
  Interviewer: User behavior should adapt recommendations within a few minutes.
• Candidate: Are we expected to design the deep learning model math parameters?
  Interviewer: No, focus on the system architecture (how data flows, where models run, how candidate generation is cached, and the database boundaries).

3. Functional Requirements

• Personalized Home Feed: Recommend items matching user historical interests.
• Related Items: Show similar/complementary items on a product detail page.
• User Interaction Logging: Capture and log clicks, skips, impressions, and purchases.
• Cold Start Handling: Support fallback strategies for new users and recently uploaded catalog products.

4. Non-Functional Requirements

• Ultra-Low Latency: Recommendation response must complete in under 100ms.
• High Ingestion Throughput: Process 120,000 interaction writes/sec without blocking reads.
• Scalability: Scale candidate indexing horizontally to support 10 million products.
• Experimentation Support: Incorporate dynamic A/B routing targets for model upgrades.

5. Capacity Estimation

1. Query Throughput (QPS)

• Home feed queries: 600 QPS average.
• Interaction Ingest QPS: 120,000 writes/sec (pushed from client tracking SDKs).

2. Ingest Storage volume

• Interaction log (user_id, item_id, event_type, timestamp, device): ~100 bytes.
• Daily events count: 120,000 QPS * 86,400s = ~10 Billion events/day.
• Daily clickstream storage: 10B * 100 bytes = 1 TB/day. (Requires distributed Hadoop/S3 storage).

3. Online Feature Store Cache Memory

To rank candidates, the model needs fast access to user historical profile features.
Assume active user profile (top 100 product categories, dynamic embedding vectors): ~500 bytes.
Active user base (last 30 days): 100 Million.
Required RAM Feature Store capacity: 100M * 500 bytes = 50 GB. (Easily cached in Redis).

6. Identify Core Components

• Recommendation Orchestrator: Coordinates the online retrieval, ranking, and filtering pipeline.
• Candidate Generation (Retrieval) Service: Filters 10M products down to ~200 items in under 15ms using Approximate Nearest Neighbor (ANN) vector search.
• Scoring & Ranking Service: Evaluates candidate items using a Deep Learning CTR model (e.g. DLRM) or GBDT.
• Re-ranking & Business Rules Service: Deduplicates, filters out purchased items, and ensures category diversity.
• Online Feature Store: Low-latency datastore containing user and product profile properties.
• Offline Training Cluster (Spark/TensorFlow): Trains recommendation models and generates vector embeddings.

7. High-Level Architecture

We separate the online inference pipeline (latency-sensitive) from the offline model training pipeline (throughput-sensitive):

8. API Design

1. Retrieve Personalized Recommendations

GET /api/v1/recommendations

Request Parameters:

• userId: "usr_4091a2de"
• pageContext: "home_feed"
• limit: 20

Response Payload (200 OK):

2. Log User Feedback

POST /api/v1/feedback

Request Payload:

9. Data Model: Feature Store Structure

To score candidate products, the ranking model pulls feature properties from the Feature Store:

10. Database Selection

We choose different databases for the online and offline loops:

• Vector Database (Pinecone/Milvus): For the online Candidate Retrieval stage. These databases index high-dimensional embedding vectors using Hierarchical Navigable Small World (HNSW) graphs, enabling K-Nearest Neighbor (k-NN) queries in <15ms.
• Online Feature Store (Redis + Cassandra): We cache hot user profiles in Redis to meet the 100ms latency budget. Cold profiles are fetched from Cassandra.
• Data Lake (Amazon S3 + ClickHouse): clickstream logs are stored in S3 for historical batch training, and ClickHouse for real-time aggregation queries.

11. Deep Dive: The Multi-Stage Recommendation Funnel

The Scoping Funnel Pipeline

1. Stage 1: Retrieval (Candidate Generation)

We reduce 10 million items to 200 candidates using fast mathematical vector calculations:
- Users and items are mapped to the same 128-dimensional embedding space.
- When a user requests their feed, the Retrieval Service pulls the user's vector and performs an Approximate Nearest Neighbor (ANN) search against the item vector index using cosine similarity:
CosineSimilarity = (A · B) / (||A|| ||B||).
- This retrieves the top 200 products closest to the user's interests in under 15ms.

2. Stage 2: Scoring & Ranking

We evaluate the 200 candidates using a deep learning CTR model (like DLRM or XGBoost):
- The service fetches detailed features (user history, device type, item price, sales speed) from the Feature Store.
- The ranking model predicts the probability of the user clicking (pCTR) and purchasing the item.
- Candidates are sorted by their final score.

3. Stage 3: Re-ranking & Filtering

Before rendering, we apply business rules:
- Deduplication: Remove items already purchased by the user.
- Category Diversity: Ensure no more than 3 consecutive items are from the same category.
- Exploration: Insert 5% random/new items to discover new interests.

12. Request Lifecycle

1. Request Ingress: Client opens the homepage. The gateway routes the request to the Recommendation Orchestrator.
2. Retrieve Candidates: The Orchestrator calls the Retrieval Service. The service fetches the user's embedding vector from Redis and queries the Vector Database, returning 200 candidate item IDs.
3. Fetch Features: The Orchestrator fetches user features and candidate item features from Redis in parallel using batch reads.
4. Predict Scores: The Orchestrator calls the Scoring Service with the combined feature vectors. The ML model predicts CTR scores and returns the sorted candidates.
5. Filter & Diversify: The Re-ranking Service filters out duplicate or purchased items, applies diversity constraints, and returns the top 20 product IDs to the client.

13. Scaling Strategy

• Sharded Vector Search: Partition HNSW vector indices across multiple data nodes. Coordinator nodes query shards in parallel, merging candidate lists.
• Offline-Online Split: Keep model training offline. Model weights are trained daily on Spark clusters and deployed to a Model Registry. The online scoring service simply downloads the static model weights and performs fast inference.

14. Bottleneck Analysis: Feature Store Latency

Candidate: Querying 20,000 keys sequentially is too slow. We optimize this using three strategies:
1. Dynamic Caching: Cache static product features (e.g. category, brand) locally in the Scoring Service memory. Product features change slowly and can be cached with a 1-hour TTL, eliminating Redis lookups for those fields.
2. Redis Pipeline Batching: For dynamic features (e.g. real-time sales speed), use Redis pipelined MGET commands. A single batch network call fetches features in parallel, reducing network roundtrip latency to <5ms.
3. Feature Serialization: Compress features into flat binary buffers (like Protobuf or FlatBuffers) to reduce data size and serialization overhead.

15. Trade-off Discussion: Two-stage Funnel

Interviewer: Why use a two-stage funnel instead of passing all 10M products directly to the deep learning scoring model?
Candidate:
- *Direct Scoring (Single Stage):* Maximizes accuracy since the deep learning model evaluates all items. However, scoring a single item takes ~1ms of CPU time. Scoring 10 million items would take 10,000 seconds (almost 3 hours), which is impossible for online retrieval.
- *Two-Stage Funnel:* Decouples retrieval from scoring. The Retrieval stage uses fast vector arithmetic (ANN) to filter out 99.9% of irrelevant items in <15ms. The scoring model only evaluates the remaining 200 items, which takes <30ms. This trade-off accepts a minor loss in retrieval precision to guarantee sub-100ms latency.

16. Failure Scenarios: Cold Start Outage

How the system handles cold start scenarios:

• New User Cold Start: A new user signs up, meaning we have no historical preferences or vector embedding to query.
  Mitigation: Fallback to popularity and editorial content. Query the top-20 trending items in the user's geolocation or display items selected by catalog curators.
• New Product Cold Start: A new product is uploaded, but has zero views/clicks, meaning the offline model training has not generated its embedding.
  Mitigation: Content-based fallback. Extract text features from the product description and map it to an initial embedding matching similar products.

17. Security Design

• Feature Poisoning Prevention: Clean interaction event logs before training models to prevent malicious users from gaming recommendations (e.g. bots click-spamming a product to boost its ranking).

18. Monitoring & Observability

• Click-Through Rate (CTR): Track user clicks relative to impressions. A drop in CTR indicates model drift or poor recommendations.
• Model Drift Metrics: Measure the mathematical divergence between the training data distribution and live query features. Alert when divergence exceeds thresholds.

19. Cost Optimization

Vector indices consume large amounts of RAM. To optimize costs, we use Quantization (Scalar Quantization) to compress float coordinates from 32-bit to 8-bit. This reduces vector index RAM requirements by 4x, saving infrastructure costs with minimal loss in retrieval accuracy.

20. Production Improvements: Near Real-time Embeddings

Instead of waiting for daily batch training jobs to update embeddings:
1. Route clickstream logs to Kafka.
2. Apache Flink processes clicks in real-time, updating the user's active session features list.
3. An online Embedding Generator combines the user's static embedding with their dynamic session features, calculating a temporary real-time embedding in Redis.
4. This allows the system to adapt recommendations within minutes when a user starts browsing a new category.

21. Interviewer Follow-up Questions

Candidate: We deploy new models using A/B testing:
1. The new model version is trained offline and registered in the Model Registry.
2. An A/B routing service splits incoming recommendation requests: 90% go to the control model (v1) and 10% to the treatment model (v2).
3. We track user engagement metrics (CTR, conversion rate) for both groups.
4. If model v2 shows a statistically significant improvement in conversion rates without increasing latency, we gradually roll it out to 100% of users.

22. Final Architecture

The complete recommendation engine architecture featuring offline pipelines, online feature stores, and A/B test routing:

23. Summary

We designed a real-time recommendation engine that personalizes product feeds under 100ms. By separating the pipeline into Candidate Retrieval (filtering 10M items to 200 in <15ms using HNSW ANN search) and Scoring (ranking candidates in <30ms using a deep learning CTR model), we met strict low-latency SLA targets. Dynamic feedback loops are processed using Kafka and Flink to update user profiles within minutes.

24. Cheat Sheet

25. Candidate Interview Evaluation