09 - Design a Notification System
📋 Jump to Takeaways🎁 A single event ("someone liked your post") fans out to push notifications, emails, and SMS across three different providers with different reliability guarantees. How do you make sure nothing gets lost?
A notification system delivers messages to users across multiple channels: push notifications (iOS/Android), SMS, and email. Think password resets, order updates, marketing campaigns. The challenge is handling millions of notifications per day across different providers with different rate limits, while never spamming users or losing critical messages.
Notification System Requirements
Notification System Functional Requirements
- Send notifications via push (APNs = Apple Push Notification service, FCM = Firebase Cloud Messaging), SMS (Twilio), and email (SendGrid)
- Respect user preferences (opt-out per channel, quiet hours)
- Support priority levels (critical: password reset, low: weekly digest)
- Schedule notifications for future delivery
- Track delivery status (sent, delivered, failed)
Notification System Non-Functional Requirements
| Metric | Target |
|---|---|
| Volume | 10 million notifications/day |
| Latency (critical) | < 30 seconds end-to-end |
| Latency (low priority) | < 5 minutes |
| Availability | 99.9% (~8.7 hrs/year, ~43 min/month, ~86 sec/day) |
| Durability | Never lose a critical notification |
Notification System Scope Exclusions
- In-app notifications (different delivery mechanism)
- Template management UI (assume templates exist)
- A/B testing of notification content
Notification System Estimation
- 10M notifications/day ÷ 86,400 = ~115/sec average, ~350/sec peak
- Per notification: ~1 KB (template + metadata)
- Storage: delivery logs at 10M/day × 500 bytes × 90 days = 450 GB
Moderate volume. The challenge is multi-channel delivery and reliability, not raw throughput.
Notification System High-Level Design
┌────────────────────────────────────────────────────────────────┐
│ NOTIFICATION SYSTEM │
├────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ API / │──►│ Validation │──►│ User Pref │ │
│ │ Event │ │ & Template │ │ Store │ │
│ │ Trigger │ │ Engine │ │ (Redis) │ │
│ └──────────┘ └──────────────┘ └──────┬───────┘ │
│ │ │
│ ┌──────────────┴──────────┐ │
│ ▼ (immediate) ▼ │
│ ┌─────────────────────┐ ┌──────────────┐ │
│ │ Message Queues │ │ Scheduled │ │
│ │ (by channel + │ │ Queue │ │
│ │ priority) │ └──────┬───────┘ │
│ └──────────┬──────────┘ │ │
│ │ ┌────────▼────────┐ │
│ ▼ │ Scheduler │ │
│ ┌───────────────────┐ │ (moves due │ │
│ │ Channel Workers │◄────│ to queues) │ │
│ │ Push │ SMS │Email│ └─────────────────┘ │
│ └───────────┬───────┘ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ Third-Party APIs │ │
│ │ APNs FCM Twilio │ │
│ │ SendGrid │ │
│ └───────────────────┘ │
└────────────────────────────────────────────────────────────────┘Notification System Deep Dive
Separate Queues Per Channel
SMS providers throttle at 100/sec. Push can handle 10,000/sec. If they share a queue, slow SMS processing blocks fast push delivery.
Separate queues also let you scale independently. Add more email workers during a marketing blast without affecting push notification latency.
Notification Priority Queues
A flood of marketing emails should never delay a password reset:
push:critical ← password reset, security alerts
push:high ← order updates
push:low ← promotions, digests
Workers always drain critical first, then high, then low.Notification Deduplication
With at-least-once delivery (queues may redeliver), the same notification could be sent twice. Use an idempotency key:
Before sending:
key = hash(user_id + event_type + event_id)
if Redis.SET_NX(key, TTL=10min) → send it
if key exists → skip (already sent)User Preferences and Quiet Hours
Check before queuing (not before sending) to avoid wasting queue capacity:
- User opted out of SMS? → don't queue an SMS
- User in quiet hours (11 PM - 7 AM local time)? → route to Scheduled Queue, deliver at 7 AM
Retry and Dead Letter
Send attempt fails (provider timeout):
→ Retry with exponential backoff (5s, 30s, 5min)
→ After 3 retries: move to dead letter queue
→ Alert team for investigation
Provider down entirely:
→ Circuit breaker opens → stop sending to that provider
→ Messages queue up → resume when provider recoversDelivery Tracking
Log every notification's lifecycle:
CREATED → QUEUED → SENT → DELIVERED (or FAILED → RETRIED → DEAD)Providers send delivery confirmations via webhooks (APNs feedback, SendGrid events). Store in a delivery log for debugging and analytics (open rates, delivery rates).
Key Takeaways
Separate queues by channel and priority
Each channel has different throughput limits. Each priority level needs independent processing. A marketing flood should never delay a security alert.
Check preferences before queuing, not before sending
If a user disabled SMS, don't waste queue capacity on an SMS that will never send. Check early, save resources.
Idempotency keys prevent duplicate sends
At-least-once delivery means duplicates will happen. Dedup with a unique key per notification before calling the provider.
Circuit breakers protect against provider outages
When Twilio is down, stop hammering it. Let messages queue. Probe periodically. Resume when healthy.
Scheduled notifications use a separate queue + scheduler
Future sends go to a scheduled store. A scheduler moves them to channel queues when their time arrives. Same pipeline from that point on.
Want the full design? The example "Notification System at Scale" covers template engines, rate limiting per provider, Kafka partitioning by user_id, and webhook ingestion.
🎁 A user follows 500 people. When they open the app, how do you assemble a personalized feed in under 200 milliseconds?