11 - Design a Chat System

🎁 WhatsApp delivers 100 billion messages per day. Every message reaches the right device within milliseconds, even when the recipient is offline. What does that architecture look like?

A chat system (like WhatsApp or Slack) delivers messages between users in real-time. The key challenges: maintaining millions of persistent connections, routing messages between servers, guaranteeing message ordering, and handling users who go offline.

Chat System Requirements

Chat System Functional Requirements

• 1:1 messaging, send and receive text messages between two users
• Group chats, up to 500 members
• Online/offline status (presence)
• Message history, scroll back through past conversations
• Read receipts, indicate when a message has been seen

Chat System Non-Functional Requirements

Chat System Scope Exclusions

• End-to-end encryption (changes the architecture significantly)
• Voice/video calls (different protocol, WebRTC)
• Message search across conversations

Chat System Estimation

• 50M DAU × 40 messages/day = 2B messages/day → ~23K msg/sec, peak ~70K/sec
• Message size: ~300 bytes → 600 GB/day storage
• WebSocket connections: 50M concurrent → ~500 servers at 100K connections each

Write-heavy with real-time delivery requirements. WebSocket connection count drives server count.

Chat System High-Level Design

┌────────────────────────────────────────────────────────────────┐
│                        CHAT SYSTEM                             │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌──────────┐        ┌──────────────────┐    ┌──────────────┐  │
│  │  Client  │◄─WSS──►│  WebSocket       │◄──►│  Connection  │  │
│  │          │        │  Gateway         │    │  Registry +  │  │
│  └──────────┘        │  (500 servers)   │    │  Presence    │  │
│                      └────────┬─────────┘    │  (Redis)     │  │
│                               │ ▲            └──────────────┘  │
│                               ▼ │                              │
│                      ┌──────────────────┐                      │
│                      │  Message Service │                      │
│                      │  (routing +      │                      │
│                      │   sequencing)    │                      │
│                      └────────┬─────────┘                      │
│                               │                                │
│                               ▼                                │
│                      ┌──────────────────┐                      │
│                      │  Message Store   │                      │
│                      │  (Cassandra)     │                      │
│                      │  partition: conv │                      │
│                      │  cluster: seq    │                      │
│                      └──────────────────┘                      │
└────────────────────────────────────────────────────────────────┘

Chat System Deep Dive

WebSocket Protocol

A WebSocket is a protocol that upgrades an HTTP connection into a persistent, bidirectional channel. Unlike HTTP (request-response), either side can send data at any time. WSS = WebSocket Secure (encrypted, like HTTPS).

HTTP is request-response. The client must ask for new messages. At 50M users, polling every second = 50M requests/sec of mostly empty responses.

WebSockets maintain a persistent bidirectional connection. Either side can send at any time. The server pushes new messages instantly without the client asking.

Connection Registry

With 500 WebSocket servers, how do you know which server a user is connected to?

Redis: user:alice → ws-server-042

When alice sends a message to bob:

1. Look up bob's server in Redis
2. If same server → deliver directly
3. If different server → publish to bob's server via Redis Pub/Sub

Message Ordering

Global ordering across all conversations is impossible at scale. Instead, guarantee order per conversation:

conversation: conv_abc123
  seq 1: alice → "hey"
  seq 2: bob   → "hi!"
  seq 3: alice → "sup?"

Each conversation has a monotonically increasing sequence number. The Message Service assigns it atomically (Redis INCR on seq:conv_abc123). Clients use sequence numbers to detect gaps and request missing messages.

Offline Delivery

User goes offline → connection removed from Registry

New message arrives for offline user:
  1. Store in Message Store (always)
  2. Send push notification (APNs/FCM)

User reconnects:
  1. Client sends last_seen_sequence per conversation
  2. Server fetches messages with seq > last_seen
  3. Delivers batch over WebSocket

Group Messages

For groups ≤ 500 members, use fan-out on write:

1. Sender sends message to group
2. Message Service writes one copy to storage
3. For each online member: route to their WebSocket server
4. For offline members: push notification

Presence Detection

The Connection Registry already tells you who's online (has an entry = connected). When a user disconnects, remove the entry and record last_seen timestamp.

Don't broadcast presence to large groups. 500 members with frequent status changes means too many updates. Fetch presence on demand when a user opens the group.

Key Takeaways

• WebSockets are essential for real-time chat

  HTTP polling wastes resources. WebSockets give instant bidirectional messaging. A connection registry (Redis) maps users to their gateway server for routing.

• Message ordering is per-conversation, not global

  Sequence numbers per conversation are simple, monotonic, and guarantee order. Clients use them to detect gaps.

• Fan-out on write works for small groups

  Deliver to all members immediately on send. Reads are simple (messages already in your inbox). Breaks down at 10K+ members.

• Offline delivery = store + push notification + replay on reconnect

  Messages are always persisted. Push notifications alert the user. On reconnect, replay everything since their last sequence number.

Want the full design? The example "Chat System at Scale" covers cross-server routing with Pub/Sub, Cassandra schema design, read receipts, media sharing, and scaling to 50M concurrent connections.

🎁 What if users upload files up to 10GB and need them synced across every device within seconds, even on unreliable connections?