Design a Notification Service

This case study simulates a realistic FAANG system design interview for designing a highly available, multi-channel notification platform. It details priority queuing, idempotency deduplication, user preferences validation, and downstream gateway failovers.

1. Present the Interview Question

2. Clarifying Questions

The candidate clarifies operational expectations and scale:

• Candidate: What is the daily and peak volume of notifications?
  Interviewer: We need to support an average of 1 billion notifications per day.
• Candidate: What is the delivery latency target for transactional vs. promotional notifications?
  Interviewer: Transactional alerts (like OTPs) must be delivered in under 5 seconds. Promotional notifications can be delayed up to a few hours during peak periods.
• Candidate: How should we handle user preferences and opt-ins?
  Interviewer: Users can opt out of specific channels (e.g., receive transactional emails but opt out of promotional push alerts).
• Candidate: How do we handle duplicate requests from upstream systems?
  Interviewer: The system must support deduplication to prevent sending the same transactional message twice.

3. Functional Requirements

• Multi-Channel Delivery: Support iOS Push (APNs), Android Push (FCM), SMS (Twilio), and Email (SendGrid/SES).
• Preference Engine: Check and enforce user notification settings before routing alerts.
• Template Management: Store, dynamic-hydrate, and render parameterized templates.
• Transactional Deduplication: Prevent double-send errors on payment receipts or OTPs.
• Campaign Delivery: Support high-volume promotional notification campaigns without blocking critical messages.

4. Non-Functional Requirements

• Strict Priority Ingress: Critical transactional alerts must bypass promotional queues.
• High Uptime: High service availability; fallback gracefully during provider outages.
• Retry Mechanism: Implement retries with exponential backoff on downstream failures.
• Idempotency: Guarantee at-least-once delivery, while striving for exactly-once to client devices.

5. Capacity Estimation

1. Throughput (QPS)

• 1 Billion notifications per day.
• Average QPS: 1,000,000,000 / 86400 seconds = ~11,600 QPS.
• Peak Ingestion QPS (Promotional Spikes): Assume 3x = ~35,000 QPS.
• Assume transactional QPS is 10% of total volume (~1,160 QPS), while promotional makes up 90%.

2. Database Log Storage Sizing

• Each log event contains: notification_id (16B), user_id (16B), channel (1B), template_id (4B), status (1B), timestamp (8B), metadata (150B) = ~200 bytes.
• Daily Log Volume: 1B * 200 bytes = ~200 GB/day.
• Monthly Log Volume: 200 GB * 30 = ~6 TB/month.
  Decision: Retain active logs in Cassandra for 30 days, then archive them to cold storage (S3 Glacier) to manage costs.

6. Core Components

• Notification API Service: Validates payloads, checks idempotency keys, and ingests events.
• Preferences & Opt-out Service: Checks user channels settings.
• Template Compilation Service: Injects parameters into template strings.
• Idempotency Cache (Redis): Detects and blocks duplicate payloads.
• Priority Message Queues (Kafka): Decouples ingestion from dispatching, splitting events into High (OTP), Medium (Transactional), and Low (Promo) queues.
• Dispatcher Workers: Consume events from Kafka and call downstream providers.
• Failover Registry: Monitors provider health and manages failovers.

7. High-Level Architecture

The High-Level setup separates API requests from worker dispatch queues to isolate traffic classes:

8. API Design

1. Send Notification Ingress

POST /api/v1/notifications

Request Payload:

Response Payload (202 Accepted):

9. Data Model

We track notification delivery states sequentially. Partitioning log tables by user_id allows users to view their notification history quickly:

10. Database Selection

Candidate:
- Notification Logs Database: We select Apache Cassandra.
Justification: Log writes are highly write-heavy (11,600 QPS average). Cassandra is optimized for fast sequential writes. Partitioning by user_id maps historical alerts to the same disk segment, keeping read queries fast.
- User Preferences Database: We choose a relational database (PostgreSQL) with a caching layer (Redis). User preferences require relational normalization (join tables mapping opt-outs per channel) and strong consistency during changes.

11. Deep Dive: Deduplication & Priority Queuing

1. Deduplication Engine (Idempotency)

To prevent duplicate notification sends (such as double OTP SMS alerts due to client network retries), the API gateway leverages a Redis idempotency lock:
1. When an ingestion request arrives containing idempotencyKey, the API service executes:
SET idempotency:idem_otp_usr90132_try1 "processing" NX PX 300000.
2. If Redis returns OK, the request is new. The service processes the request and writes it to Kafka.
3. If Redis returns null, the request is a duplicate. The service checks the status: if it is "processing", it blocks the duplicate call or returns the cached response, preventing double sending.

2. Priority Queue Layout

12. Request Lifecycle

1. Ingestion: Upstream payment service calls POST /api/v1/notifications with transactional parameters. The API Service verifies the idempotency key against Redis.
2. Preference Check: The API Service queries the User Preference Cache (Redis/PostgreSQL). If the user has opted out of emails, the system aborts processing and returns a success response.
3. Queue Routing: The API Service identifies the priority (e.g. HIGH for transactional OTPs) and writes the event payload to the Kafka High Priority topic.
4. Worker Consumption: High-priority Dispatcher Workers poll the High topic, fetch template strings from the Template Service, and compile the final text using the payload params.
5. Gateway Delivery: The worker queries the Failover Registry to identify the primary provider (e.g., SendGrid for email). It issues an HTTP call to SendGrid, writes a delivery status log to Cassandra, and returns the response.

13. Scaling Strategy

• Kafka Partition Sharding: Partition Kafka topics using the user_id as the partitioning key. This guarantees that notifications for a specific user are processed sequentially by the same worker thread, preventing order violations (e.g., ensuring a "Verification Code" is delivered before an "Expired Code" alert).
• Decoupled Dispatcher Workers: Dispatchers are stateless container instances grouped by priority. High-priority worker pools are sized to ensure that latency-sensitive OTPs are never delayed by slow-moving promotional campaigns.

14. Bottleneck Analysis: Downstream Provider Failure

Candidate: We implement a Failover Registry and Circuit Breakers on the workers:
1. Dispatcher API calls to SendGrid are wrapped in a Circuit Breaker (e.g. using Resilience4j).
2. If SendGrid returns a 5xx error or connections timeout, the worker catches the exception and updates the Failover Registry.
3. The registry dynamically switches the active provider for the channel to a backup provider (e.g., Mailgun or Amazon SES).
4. The worker retries the failed message using the backup provider, ensuring uninterrupted delivery.

15. Trade-off Discussion: Priority Queues

Interviewer: Why use three priority queues instead of one queue sorted by priority?
Candidate:
- *Sorted Priority Queue:* Keeps all events in a single datastore, sorted by priority. However, sorting at scale is expensive (requiring heap structures or index scans). As queue sizes grow to millions of messages during marketing campaigns, sorting creates high CPU overhead and insertion bottlenecks.
- *Separate Queues:* Decouples traffic class streams completely. Workers pull from queues independently. During a marketing campaign, the High queue remains empty, meaning OTPs bypass the backlog with zero sorting overhead. This trades a small increase in infrastructure complexity for O(1) ingestion and guaranteed low-latency delivery.

16. Failure Scenarios

How the system handles corrupt messages:

• Poison Pill Messages: A message with corrupt template parameters crashes the worker thread. When the worker restarts, it pulls the same message, creating a crash loop.
  Mitigation: Configure a Dead Letter Queue (DLQ). If a message fails processing 3 times, the worker moves it to the DLQ, updates the notification status to "failed", and commits the Kafka offset to resume processing the next message.
• Worker Crash during Ingestion: A worker crashes after pulling a message from Kafka but before sending it.
  Mitigation: Configure manual offset commits. The worker commits the Kafka offset only after receiving a successful delivery response from the downstream provider.

17. Security Design

• Webhook Signature Verification: Downstream providers (like SendGrid or Twilio) publish delivery receipt events back to our webhooks. We validate the HMAC SHA256 signatures on these incoming requests to prevent spoofed status updates.
• Secrets Vault: Secure API keys for gateways (APNs, Twilio, SendGrid) inside a secrets vault (AWS Secrets Manager or HashiCorp Vault), rotating keys periodically.

18. Monitoring & Observability

• Delivery Success Rate: Percentage of successfully delivered notifications per channel/provider.
• Kafka Consumer Lag: The number of queued messages waiting to be processed. A spike in lag triggers autoscaling for worker nodes.

19. Cost Optimization

SMS notifications are expensive. To minimize costs:
1. Enforce channel fallback ordering: if a user is active on the mobile app, send a cheap Push Notification first.
2. If the push notification delivery fails (tracked via receipt feedback), fall back to sending a more expensive SMS after a 2-minute delay.

20. Production Improvements

Implement a Smart Dynamic Routing Router. The router monitors delivery latencies and success rates for different SMS providers (e.g. Twilio vs. Plivo) per country. It dynamically routes SMS traffic to the cheapest provider that meets our target latency SLO in each region.

21. Interviewer Follow-up Questions

Candidate: Yes. We check preferences twice during the lifecycle:
1. At Ingestion: The API Service checks the preference cache and rejects opt-out messages before they queue.
2. At Dispatch: Because campaigns can queue in Kafka for hours, the user might change their preferences during that time. The Dispatcher Worker performs a final fast check against Redis before calling the provider API. If the user has opted out in the meantime, the worker discards the message, ensuring opt-outs are respected immediately.

22. Final Architecture

The complete optimized notification service architecture featuring priority queues, failover gateways, and DLQ handling:

23. Summary

We designed a high-throughput notification platform supporting 1 billion notifications/day. By decoupling ingestion from dispatch using priority Kafka queues, we isolated critical OTP traffic from promotional backlogs. Transactional deduplication is handled using Redis idempotency locks, while downstream gateway outages are managed using dynamic failover routing.

24. Cheat Sheet

25. Candidate Interview Evaluation