Notification System at Scale

Step 1: Notification System Requirements

Notification System Functional Requirements

Multi-channel delivery: Push notifications (iOS via APNS, Android via FCM), SMS (Twilio), Email (SendGrid/SES)
User preferences: Opt-in/out per channel, quiet hours, frequency caps
Templates: Reusable message templates with variable substitution
Scheduling: Send immediately or schedule for a future time
Priority levels: Critical (password reset), High (order update), Medium (promotion), Low (weekly digest)
Tracking: Delivery status, open rates, click-through for each notification

Notification System Non-Functional Requirements

Metric	Target
Volume	10M notifications/day
Latency (high priority)	< 30 seconds end-to-end
Latency (low priority)	< 5 minutes
Delivery guarantee	At-least-once
Delivery success rate	99.9%
Availability	99.95% uptime (maximum allowable downtime limits - Daily: 43 seconds, Monthly: 21 minutes and 55 seconds, Yearly: 4 hours and 23 minutes)
Data retention	90 days for delivery logs

Step 2: Notification System Estimation

Notification System Throughput

Daily volume:     10M notifications/day
Average rate:     10,000,000 / 86,400 ≈ 116 notifications/sec
Peak rate (10x):  ~1,200 notifications/sec

Message Sizes

Channel	Payload Size	Notes
Push (APNS/FCM)	2-4 KB	Title + body + metadata
SMS	160-480 bytes	1-3 segments
Email	10-50 KB	HTML body + headers

Notification System Storage

Templates:        ~10,000 templates × 5 KB = 50 MB (negligible)
Delivery logs:    10M/day × 500 bytes × 90 days = 450 GB
Scheduled queue:  ~1M pending × 1 KB = 1 GB
User preferences: 50M users × 200 bytes = 10 GB (fits in Redis)

Third-Party Rate Limits

Provider	Rate Limit	Our Budget
APNS	No hard limit (but throttles)	Batch at 5,000/sec
FCM	500 msg/sec per project (expandable)	Request quota increase
Twilio	100 SMS/sec (default)	Scale with subaccounts
SendGrid	10,000 emails/sec (pro plan)	Comfortable headroom

Step 3: Notification System High-Level Design

┌────────────────────────────────────────────────────────────────────┐
│                        NOTIFICATION SYSTEM                         │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  ┌──────────┐   ┌──────────────┐   ┌──────────────┐                │
│  │ API /    │   │  Validation  │   │  User Pref   │                │
│  │ Event    │──▶│  & Template  │──▶│  Store       │                │
│  │ Trigger  │   │  Engine      │   │  (Redis)     │                │
│  └──────────┘   └──────────────┘   └──────┬───────┘                │
│                                           │                        │
│                            ┌──────────────┴──────────────┐         │
│                            │                             │         │
│                            ▼ (immediate)                 ▼         │
│              ┌──────────────────────┐         ┌──────────────┐     │
│              │    Message Queues    │         │  Scheduled   │     │
│              │                      │◀───┐    │  Queue       │     │
│              │ ┌─────┐┌─────┐┌────┐ │    │    └──────┬───────┘     │
│              │ │Push ││SMS  ││Mail│ │    │           │             │
│              │ │Queue││Queue││Que │ │    │           ▼             │
│              │ └─────┘└─────┘└────┘ │    │    ┌──────────────┐     │
│              └──────────┬───────────┘    └────│  Scheduler   │     │
│                         │                     │  (cron jobs) │     │
│                         ▼                     └──────────────┘     │
│              ┌───────────────────────┐                             │
│              │  Notification Workers │                             │
│              │ ┌────┐ ┌────┐ ┌─────┐ │                             │
│              │ │Push│ │SMS │ │Email│ │                             │
│              │ │Wkr │ │Wkr │ │Wkr  │ │                             │
│              │ └──┬─┘ └──┬─┘ └──┬──┘ │                             │
│              └────┼──────┼──────┼────┘                             │
│                   │      │      │                                  │
│                   ▼      ▼      ▼                                  │
│              ┌───────────────────────┐                             │
│              │    Rate Limiter       │                             │
│              │ (per user & provider) │                             │
│              └───────────┬───────────┘                             │
│                          │                                         │
│                          ▼                                         │
│              ┌────────────────────────┐                            │
│              │  Third-Party Providers │                            │
│              │  APNS FCM Twilio SES   │                            │
│              └───────────┬────────────┘                            │
│                          │ (webhooks/callbacks)                    │
│                          ▼                                         │
│              ┌───────────────────────┐                             │
│              │     Delivery Log      │                             │
│              │  (Postgres + TS DB)   │                             │
│              └───────────────────────┘                             │
└────────────────────────────────────────────────────────────────────┘

Notification System Component Responsibilities

Component	Role
API/Event Trigger	Accepts notification requests from services or event bus
Validation & Template Engine	Renders templates, validates payload, checks preferences. Routes to Message Queues (immediate) or Scheduled Queue (future send_at)
User Preference Store	Fast lookup of channel opt-in/out, quiet hours
Scheduled Queue	Holds notifications with a future send_at until they are due
Scheduler	Polls the Scheduled Queue, moves due notifications into Message Queues
Message Queues	Buffers work, separates by channel and priority
Notification Workers	Channel-specific workers that call third-party APIs
Rate Limiter	Enforces per-user and per-provider rate limits
Scheduler	Moves scheduled notifications to queues at send time
Delivery Log	Records status, enables retry, powers analytics

Step 4: Notification System Deep Dive

Message Queue Design

Separate queues by priority and channel for independent scaling:

Queues:
  push:critical    ← 3 consumers, poll every 100ms
  push:high        ← 3 consumers, poll every 500ms
  push:medium      ← 2 consumers, poll every 1s
  push:low         ← 1 consumer, batch every 5s

  sms:critical     ← 2 consumers
  sms:high         ← 2 consumers
  sms:medium       ← 1 consumer
  sms:low          ← 1 consumer (batched)

  email:critical   ← 2 consumers
  email:high       ← 2 consumers
  email:medium     ← 1 consumer
  email:low        ← 1 consumer (batched/digest)

Why separate queues? A flood of low-priority marketing emails should never delay a high-priority password reset push notification. Each queue can be scaled independently based on backlog depth.

Technology choice: Kafka (for ordering guarantees and replay) or SQS (for simplicity and auto-scaling). Use Kafka topic partitions keyed by user_id to maintain per-user ordering.

Notification System Deduplication

Duplicate notifications destroy user trust. Use idempotency keys to prevent re-sends:

Idempotency key = hash(event_type + user_id + content_hash + timestamp_bucket)

Example: hash("order_shipped" + "user_123" + "abc123" + "2026-05-19T10:00")

Implementation:

Before enqueuing, check if key exists in Redis (TTL = 24 hours)
If exists → skip (already sent or in-flight)
If not → set key, enqueue message
On successful delivery → keep key until TTL expires
On permanent failure → remove key (allow retry from source)

This prevents duplicates from:

Upstream services firing the same event twice
Queue redelivery after consumer crash
Retry storms during partial outages

Notification System Retry Strategy

Different channels have different failure modes:

Channel	Retry Policy	Max Retries	Backoff
Push (APNS/FCM)	Exponential	5	1s, 5s, 30s, 2m, 10m
SMS (Twilio)	Exponential	3	10s, 60s, 5m
Email (SendGrid)	Exponential	5	30s, 2m, 10m, 1h, 6h

Retry logic:

retry_delay = base_delay × 2^(attempt - 1) + random_jitter

For push:  1s × 2^0 = 1s, 1s × 2^1 = 2s, ... capped at 10m
Jitter:    ±20% to prevent thundering herd

Non-retryable errors (skip immediately):

Invalid device token (APNS 410 Gone)
Invalid phone number (Twilio 21211)
Unsubscribed email (SendGrid bounce)

Dead Letter Queue (DLQ): After max retries exhausted, move to DLQ. Alert on DLQ depth. Manual review or batch retry after provider recovery.

User Preference Checking

Preferences must be checked on every notification. This is a hot path:

Storage: Redis hash per user
Key:     user_prefs:{user_id}
Fields:  push_enabled, sms_enabled, email_enabled,
         quiet_start, quiet_end, timezone,
         frequency_cap_push, frequency_cap_sms

Lookup time: < 1ms (Redis)
Cache hit rate: 99%+ (most users don't change prefs often)

Check order (fail fast):

Is user globally unsubscribed? → Drop
Is this channel enabled for user? → Drop if not
Is user in quiet hours? → Reschedule for quiet_end
Has user exceeded frequency cap? → Drop or downgrade priority
Proceed to send

Write-through cache: When user updates preferences via API, update both Postgres (source of truth) and Redis (cache) synchronously.

Template Rendering

Templates enable consistent messaging without hardcoding content in every service:

# Template: order_shipped
channels:
  push:
    title: "Your order is on its way! 📦"
    body: "{{item_name}} shipped via {{carrier}}. Track: {{tracking_url}}"
  sms:
    body: "Your {{item_name}} shipped! Track at {{tracking_url}}"
  email:
    subject: "Shipment Confirmation - Order #{{order_id}}"
    template_id: "d-abc123"  # SendGrid dynamic template
    variables:
      item_name: "{{item_name}}"
      tracking_url: "{{tracking_url}}"
      delivery_date: "{{estimated_delivery}}"

localization:
  es:
    push:
      title: "¡Tu pedido está en camino! 📦"
      body: "{{item_name}} enviado por {{carrier}}. Rastrear: {{tracking_url}}"

Rendering pipeline:

Look up template by template_id + locale
Substitute variables (Mustache/Handlebars style)
Validate output (length limits per channel: push ≤ 4KB, SMS ≤ 480 chars)
Attach rendered payload to queue message

Localization: Store translations per template. Fall back to en if user's locale not available.

Delivery Tracking

Track every notification through its lifecycle:

States: CREATED → QUEUED → SENT → DELIVERED → READ → FAILED

delivery_log table:

notif_id	user_id	channel	status	provider	updated_at
n_abc123	user_456	push	DELIVERED	APNS	2026-05-19..
n_def789	user_456	email	SENT	SendGrid	2026-05-19..

How we know delivery status:

APNS: Responds synchronously (200 = accepted). No delivery receipt. Assume delivered unless feedback service reports uninstall.
FCM: Returns message_id on success. Delivery receipts available via FCM Data API.
Twilio SMS: Webhook callback with status (queued → sent → delivered/failed).
SendGrid Email: Event webhook (processed → delivered → opened → clicked → bounced).

Webhook ingestion: Dedicated webhook receiver service → writes status updates to delivery log. Idempotent (same status update applied multiple times = no-op).

Rate Limiting

Two layers of rate limiting:

Per-user rate limiting (protect users from spam):

Rule: Max 5 push notifications per hour per user
Rule: Max 2 SMS per day per user (unless critical)
Rule: Max 10 emails per day per user

Implementation: Redis sliding window counter
Key: rate:{user_id}:{channel}:{window}

Per-provider rate limiting (respect API limits):

Rule: Max 5,000 APNS calls/sec
Rule: Max 500 FCM calls/sec
Rule: Max 100 Twilio SMS/sec
Rule: Max 10,000 SendGrid emails/sec

Implementation: Token bucket algorithm
- Refill rate = provider limit
- Bucket size = 2× limit (allow short bursts)
- If bucket empty → back-pressure to queue (stop consuming)

Priority override: Critical notifications (password reset, security alerts) bypass per-user rate limits but still respect provider limits.

Failure Handling

Scenario: APNS is down for 10 minutes

Timeline:
T+0s:    APNS returns 503. Worker retries after 1s.
T+1s:    Still 503. Circuit breaker counter: 1/5.
T+5s:    5 consecutive failures. Circuit breaker OPENS.
T+5s:    Push queue consumers PAUSE. Messages accumulate.
T+35s:   Circuit breaker half-open. Send 1 probe request.
T+35s:   Still failing. Circuit breaker stays OPEN. Wait 60s.
T+95s:   Half-open probe. Still failing. Back off to 2m.
T+10m:   APNS recovers. Probe succeeds. Circuit breaker CLOSES.
T+10m:   Consumers resume. Backlog drains at normal rate.
T+12m:   Backlog cleared. All queued messages delivered.

Dead Letter Queue (DLQ):

Messages that fail after all retries → DLQ
DLQ messages are NOT automatically retried
Alert fires when DLQ depth > 100
Ops team reviews: fix issue, then replay DLQ messages
Permanent failures (invalid token, unsubscribed) → log and discard

Fallback channels:

If push fails after all retries AND notification is high priority → fall back to SMS
If SMS fails → fall back to email
Fallback chain defined per notification template

Key Takeaways

Separate queues by priority and channel

A marketing email flood should never delay a security alert push notification. Independent scaling per queue prevents cross-channel interference.
Idempotency keys are essential

At-least-once delivery means your system WILL see duplicates. Dedup at the enqueue layer using deterministic keys to prevent users from receiving the same notification twice.
Respect both user limits and provider limits

Per-user rate limiting protects user experience (no spam), per-provider rate limiting prevents API throttling and account suspension. Use token buckets for providers, sliding windows for users.
Circuit breakers prevent cascade failures

When a provider is down, stop hammering it. Let messages queue up, probe periodically, and resume when healthy. This protects both your system and the provider relationship.
Design for observability from day one

Track every notification through its full lifecycle (created → queued → sent → delivered → read). Webhook ingestion from providers closes the feedback loop and powers retry decisions, analytics, and debugging.
Template-driven content scales teams

Separating notification content from notification infrastructure lets product teams update messaging without deploying code. Add localization support early. Retrofitting it is painful.