Chat System at Scale
📋 Jump to TakeawaysStep 1: Chat System Requirements
Chat System Functional Requirements
- 1:1 messaging: send and receive text messages between two users
- Group chats: up to 500 members per group, with admin controls
- Online/offline status: show whether a user is currently active
- Read receipts: indicate when a message has been seen by the recipient
- Message history: scroll back through past conversations
- Media sharing: send images, videos, and files
Chat System Non-Functional Requirements
- 50M DAU (daily active users)
- Real-time delivery: messages arrive in under 500ms for online users
- Message ordering: guaranteed within a conversation (not globally)
- Durability: messages stored forever, zero data loss
- Availability: 99.99% uptime (~52 minutes downtime/year)
Step 2: Chat System Estimation
Messages Per Day
- 50M DAU × 40 messages/user/day = 2 billion messages/day
- Peak QPS: 2B / 86,400 ≈ 23,000 msg/sec average, ~70,000 msg/sec peak (3× average)
Chat System Storage
- Average message size: ~200 bytes (text) + 100 bytes metadata = 300 bytes/message
- Daily storage: 2B × 300 bytes = 600 GB/day
- Yearly storage (text only): 600 GB × 365 = ~219 TB/year
- With media (assume 10% of messages have media, avg 200KB): 2B × 0.1 × 200KB = 40 TB/day media
WebSocket Connections
- 50M concurrent connections (one per active device)
- At ~100K connections per server → 500 WebSocket servers
Chat System Bandwidth
- Inbound: 70K msg/sec × 300 bytes = 21 MB/sec (text only)
- With media bursts: significantly higher, handled by CDN offload
Step 3: Chat System High-Level Design
┌────────────────────────────────────────────────────────────────────┐
│ CHAT SYSTEM │
├────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────────────┐ ┌──────────────┐ │
│ │ Mobile │◄─WSS──►│ │◄──────►│ Connection │ │
│ │ Client │ │ WebSocket │ │ Registry + │ │
│ └────┬─────┘ │ Gateway │ │ Presence │ │
│ │ │ (500 servers) │ │ (Redis) │ │
│ ┌────┴─────┐ │ │ │ │ │
│ │ Web │◄─WSS──►│ │ │ user→server │ │
│ │ Client │ └────────┬─────────┘ │ + last_seen │ │
│ └────┬─────┘ │ ▲ └──────────────┘ │
│ │ │ │ (deliver to │
│ │ ▼ │ recipient) │
│ │ ┌──────────────────┐ │
│ │ │ Message Service │◄───┐ │
│ │ │ (routing + │ │ │
│ │ │ sequencing) │ │ │
│ │ └──┬─────┬────┬────┘ │ │
│ │ │ │ │ │ │
│ │ │ │ └─────────┼────┐ │
│ │ │ ▼ │ ▼ │
│ │ │ ┌──────────────┐ │ ┌────────────────┐ │
│ │ │ │Group Service │──┘ │ Push Notif. │ │
│ │ │ │(PostgreSQL) │ │ Service │ │
│ │ │ │members, roles│ │ (APNs/FCM) │ │
│ │ │ └──────────────┘ │ offline users │ │
│ │ ▼ └────────────────┘ │
│ │ ┌──────────────────┐ │
│ │ │ Message Queue │ │
│ │ │ (Kafka) │ │
│ │ │ partitioned by │ │
│ │ │ conversation_id │ │
│ │ └────────┬─────────┘ │
│ │ ▼ │
│ │ ┌──────────────────┐ │
│ │ │ Message Store │ │
│ │ │ (Cassandra) │ │
│ │ │ partition: conv │ │
│ │ │ cluster: seq │ │
│ │ └──────────────────┘ │
│ │ │
│ │ ┌──────────────────┐ │
│ └───────►│ Media Service │ │
│ (pre-signed │ (S3 + CDN) │ │
│ upload) └──────────────────┘ │
└────────────────────────────────────────────────────────────────────┘Key connections:
- Gateway ↔ Message Service: bidirectional: receives sent messages, delivers to recipients
- Message Service → Group Service: fetches member list; response feeds back into routing
- Message Service → Push Notification: called when recipient is offline (checked via Connection Registry)
- Message Service → Kafka → Message Store: persistence path
- Clients → Media Service: direct upload via pre-signed URLs (bypasses Gateway)
Message send flow (1:1):
Sender ──WSS──► Gateway ──► Message Service ──► Kafka ──► Message Store
│
├── lookup recipient in Connection Registry
│
├── online? ──► recipient's Gateway ──WSS──► Recipient
│
└── offline? ──► Push Notification ServiceMessage send flow (group):
Sender ──► Message Service ──► Group Service (get member list)
│
├── Kafka (persist)
│
└── for each member:
├── online? ──► route to their Gateway
└── offline? ──► push notificationConnection Management
- Client opens a WebSocket connection to the gateway (via load balancer)
- Gateway registers the mapping
user_id → server_idin the Connection Registry (Redis) - All messages for that user are routed to the correct gateway server
Message Routing (1:1)
- Sender's client sends message over WebSocket
- Gateway forwards to Message Service
- Message Service persists to Message Store (via Kafka for durability)
- Message Service looks up recipient's server in Connection Registry
- If online: push message to recipient's gateway server → deliver via WebSocket
- If offline: store in undelivered queue + trigger push notification
Step 4: Chat System Deep Dive
WebSocket Connection Management
The Connection Registry (Redis) maintains a real-time mapping:
user:12345 → { server: "ws-server-042", connected_at: 1716000000 }Challenges:
- A user disconnects unexpectedly → server detects via heartbeat timeout (30s), removes entry
- A user reconnects to a different server → old entry is overwritten atomically
- Multi-device support → store a set of connections per user:
user:12345:devices → [phone, laptop]
Scaling: Partition the registry by user_id hash. Each gateway server maintains a local cache of its own connections and syncs to Redis.
Message Routing: Same Server vs. Different Server
Same server: Gateway delivers directly from sender's connection handler to recipient's connection handler, no network hop.
Different server: Use a pub/sub layer:
┌──────────┐ publish ┌───────────────┐ subscribe ┌──────────┐
│ WS Server│───────────────►│ Redis Pub/Sub │───────────────►│ WS Server│
│ (A) │ │ or Kafka │ │ (B) │
└──────────┘ └───────────────┘ └──────────┘- Each WS server subscribes to a channel named after itself (e.g.,
ws-server-042) - To route a message: look up recipient's server in registry → publish to that server's channel
- Why Kafka over Redis Pub/Sub for large scale: Kafka provides durability if the target server is temporarily unreachable; Redis Pub/Sub is fire-and-forget
Message Ordering
Global ordering is impossible at scale. Instead, guarantee ordering per conversation:
conversation_id: "conv_abc123"
messages:
- { seq: 1, sender: "alice", text: "hey", ts: 1716000001 }
- { seq: 2, sender: "bob", text: "hi!", ts: 1716000002 }
- { seq: 3, sender: "alice", text: "sup?", ts: 1716000003 }- Each conversation has a monotonically increasing sequence number
- Message Service assigns the next sequence atomically (Redis INCR on
seq:conv_abc123) - Clients use sequence numbers to detect gaps and request missing messages
- Kafka topic partitioned by
conversation_idensures ordered processing per conversation
Group Messaging
Fan-out on write (chosen for groups ≤ 500 members):
- Sender sends message to group
- Message Service writes one copy to the group's message store
- For each online member: route the message to their WS server
- For offline members: increment unread counter + queue push notification
Why fan-out on write for small groups:
- Latency is predictable, each member gets the message immediately
- Read path is simple, no assembly required
- 500 members × 70K groups active/sec is manageable
Fan-out on read would be better for broadcast channels with 100K+ subscribers (not our case).
Offline Message Delivery
User goes offline:
1. Connection Registry entry expires (heartbeat timeout)
2. New messages → stored in "undelivered" queue per user
User reconnects:
1. Client sends last_seen_sequence per conversation
2. Server fetches all messages with seq > last_seen_sequence
3. Delivers batch over WebSocket
4. Clears undelivered queue- Push notifications sent via APNs/FCM for high-priority messages
- Batch notifications to avoid spamming (e.g., "5 new messages from Alice")
- Undelivered queue stored in Redis (TTL 30 days) with overflow to persistent store
Message Storage
Schema (Cassandra):
Partition key: conversation_id
Clustering key: sequence_number (ascending)
┌─────────────────┬─────┬────────┬──────────┬───────────┐
│ conversation_id │ seq │ sender │ content │ timestamp │
├─────────────────┼─────┼────────┼──────────┼───────────┤
│ conv_abc123 │ 1 │ alice │ "hey" │ 17160... │
│ conv_abc123 │ 2 │ bob │ "hi!" │ 17160... │
│ conv_abc123 │ 3 │ alice │ "sup?" │ 17160... │
└─────────────────┴─────┴────────┴──────────┴───────────┘Why Cassandra:
- Optimized for write-heavy workloads (2B writes/day)
- Partition by conversation_id → all messages for a chat on same node
- Time-sorted within partition → efficient range queries for history
- Linear horizontal scaling
Tiered storage: Hot data (last 30 days) on SSD, cold data on HDD/S3 for cost efficiency.
Read Receipts
Store the last read position per user per conversation:
┌─────────┬─────────────────┬───────────────────┐
│ user_id │ conversation_id │ last_read_seq │
├─────────┼─────────────────┼───────────────────┤
│ bob │ conv_abc123 │ 3 │
│ alice │ conv_abc123 │ 2 │
└─────────┴─────────────────┴───────────────────┘- Client sends "read up to seq X" when user views messages
- Server updates
last_read_seqand notifies the sender - For groups: aggregate read receipts (e.g., "read by 3 of 5") rather than sending individual updates for each member
- Debounce updates. Don't send a receipt for every single message during rapid scrolling
Presence / Online Status
Heartbeat-based approach:
- Client sends heartbeat every 30 seconds over WebSocket
- Server updates Redis key with TTL:
presence:user_123 → online (TTL 35s) - If heartbeat stops → key expires → user is offline
Broadcasting presence:
- For 1:1 chats: notify the other user directly when status changes
- For small groups (<50 members): broadcast to all members
- For large groups (50-500 members): do NOT broadcast. Fetch on demand when user opens the group
- Use a "last seen" timestamp instead of real-time status for large groups
Optimization: Batch presence updates. Instead of pushing every status change immediately, collect changes over a 5-second window and push a batch update.
Key Takeaways
WebSockets are essential for real-time chat
HTTP polling wastes resources at 50M concurrent users. A persistent WebSocket connection enables instant bidirectional messaging. A connection registry (Redis) maps users to their gateway servers for cross-server routing.
Message ordering is per-conversation, not global
Use monotonic sequence numbers per conversation and partition Kafka/storage by conversation_id. This maintains order without a global bottleneck. Clients use sequence numbers to detect gaps and request missing messages.
Fan-out on write works for groups up to ~500 members
Deliver messages to all members immediately on send. Reads become simple lookups. For massive broadcast channels (100K+ subscribers), switch to fan-out on read to avoid write amplification.
Offline delivery requires a separate queue
Store undelivered messages, send push notifications, and replay missed messages on reconnect using the client's last-seen sequence number. Batch notifications to avoid spamming ("5 new messages from Alice").
Presence is expensive to broadcast
Use heartbeats with TTL-based expiry. Avoid broadcasting status changes to large groups. Fetch presence on demand instead. Brief network blips shouldn't show users as offline; add a grace period before marking offline.
Cassandra fits the write-heavy pattern
Partition by conversation_id for data locality, cluster by sequence number for efficient history queries. Tier storage (SSD for hot data, HDD/S3 for cold) controls cost at 200+ TB/year scale.