04 - Design a Logging System

🎁 Your system produces 50,000 log events per second. Where do they all go, and how do you search them later?

A centralized logging system collects log entries from hundreds of application servers, stores them durably, and lets engineers search and filter them. Think Datadog Logs, Splunk, or the ELK stack (Elasticsearch, Logstash, Kibana). This problem introduces write-heavy systems, the opposite of everything we've built so far.

Logging System Requirements

Logging System Functional Requirements

• Ingest log entries from multiple application servers
• Store logs with timestamp, severity level, service name, and message
• Search logs by time range, severity, service, and keyword
• Retain logs for 30 days, then auto-delete
• Dashboard showing log volume and error rates

Logging System Non-Functional Requirements

Logging System Scope Exclusions

• Metrics/APM (separate system, different access patterns)
• Alerting (built on top of logging, not part of core)
• Log parsing/structuring at ingest (assume structured JSON logs)

Logging System Estimation

Logging System Traffic

• 50,000 logs/sec peak
• Daily volume: 50K × 86,400 = 4.3 billion logs/day
• Average: ~30K logs/sec (peak is 50K)

This is a write-heavy system. Reads (searches) are infrequent compared to writes.

Logging System Storage

• Per log: 500 bytes
• Daily: 4.3B × 500 bytes = 2.15 TB/day
• 30-day retention: 2.15 × 30 = ~65 TB total

Logging System Bandwidth

• Ingest: 50K logs/sec × 500 bytes = 25 MB/sec
• This is significant. The system must handle sustained high-throughput writes

Logging System Key Insight

This is the first write-heavy system in the course. Pastebin, To-Do, and URL Shortener were all read-heavy or low-traffic. Here, the challenge is ingesting 50K events/sec without dropping data, while still supporting search.

Logging System High-Level Design

┌─────────────────────────────────────────────────────────────────┐
│                       LOGGING SYSTEM                            │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌────────────┐     ┌──────────────┐     ┌──────────────────┐   │
│  │ App Server │────►│              │     │  Elasticsearch   │   │
│  │ App Server │────►│  Kafka       │────►│                  │   │
│  │ App Server │────►│  (buffer)    │     │  inverted index  │   │
│  │ ...        │────►│              │     │  time-partitioned│   │
│  │ (500 svrs) │     └──────────────┘     └────────┬─────────┘   │
│  └────────────┘                                   │ ▲           │
│                                                   │ │ (query)   │
│                                                   │ │           │
│  ┌────────────────┐                      ┌────────┴─┴────────┐  │
│  │  Cleanup Job   │─── (drop old ───────►│  Search API       │  │
│  │  (daily cron)  │     indices)         │  (query logs)     │  │
│  └────────────────┘                      └───────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Why a message queue between app servers and storage?

Without Kafka, app servers write directly to Elasticsearch. If Elasticsearch is slow or down:

• Logs are dropped (data loss)
• App servers block waiting for writes (latency impact on your actual product)

Kafka decouples producers from consumers:

• App servers fire-and-forget to Kafka (fast, never blocks)
• Kafka buffers during Elasticsearch slowdowns
• If Elasticsearch goes down for 5 minutes, logs queue up and get processed when it recovers

Logging System Deep Dive

Log Entry Structure

{
  "timestamp": "2026-05-27T10:15:32.456Z",
  "level": "ERROR",
  "service": "payment-api",
  "host": "prod-web-042",
  "message": "Failed to charge card: timeout after 5000ms",
  "trace_id": "abc-123-def",
  "metadata": {
    "user_id": "user_789",
    "amount": 49.99
  }
}

Key fields for querying:

• timestamp: always filter by time range first (narrows the search space)
• level: filter by ERROR, WARN, INFO, DEBUG
• service: "show me only payment-api logs"
• message: full-text keyword search
• trace_id: correlate logs across services for a single user request (e.g., find all logs from gateway → payment → notification for one checkout)

Elasticsearch for Log Storage

What is Elasticsearch? A search engine that also stores data. Think of it as a specialized database optimized for full-text search and filtering, not for transactions or relationships.

It stores JSON documents (not rows), queries with a JSON DSL (not SQL), and scales horizontally by sharding across nodes. For logging, it's both the storage and the search engine in one.

Why not PostgreSQL? At 50K inserts/sec and 65 TB, PostgreSQL would struggle. Full-text search on 4.3B rows/day is impractical with SQL LIKE queries. Elasticsearch is purpose-built for this.

Why not just S3? S3 is cheap storage but has no search capability. You'd need to download and scan files. Acceptable for archival (logs older than 30 days) but not for active searching.

Time-Based Partitioning

Create one index per day:

logs-2026-05-25
logs-2026-05-26
logs-2026-05-27  ← today (active writes)

Benefits:

• Queries for "last 1 hour" only search today's index (not all 65 TB)
• Deleting old logs = drop the entire index (instant, no row-by-row delete)
• Each day's index can have different replica counts (today: 2 replicas, old: 1 replica)

Cleanup: A cron job runs daily and deletes indices older than 30 days. One API call per index. No scanning, no garbage collection.

Ingestion Pipeline

App Server → Log Agent → Kafka topic "logs" → Consumer group → Elasticsearch bulk API

How do app servers send logs to Kafka? Two approaches:

Sidecar is more common in production. Apps just write to stdout (standard logging), and a lightweight agent handles forwarding. If the agent crashes, logs buffer in the file until it restarts.

Why Kafka topics partitioned by service?

• Logs from the same service land on the same partition
• Consumers can be assigned per-service for isolation
• If one service floods logs (10× normal), it doesn't block other services

Bulk writes to Elasticsearch:

• Don't write one log at a time (too many round trips)
• Buffer 1,000 logs or 1 second (whichever comes first)
• Send as a single bulk request for much higher throughput

Handling Backpressure

What if logs come in faster than Elasticsearch can index?

Normal: 50K logs/sec in, 50K logs/sec indexed → balanced
Spike:  200K logs/sec in, 50K logs/sec indexed → Kafka buffers the excess

Kafka retention: 24 hours
  → Even a 4-hour Elasticsearch outage doesn't lose data
  → Consumers catch up when capacity returns

What if Kafka itself is full? App servers should have a local buffer (in-memory ring buffer, 10 seconds). If that fills too, drop DEBUG/INFO logs first. Keep ERROR logs. This is graceful degradation: losing some debug logs is acceptable; losing error logs is not.

Where can data loss happen? Only in the app server's local buffer. If both the buffer is full AND Kafka is unreachable, logs are dropped. This matches our requirement: "acceptable to lose a few seconds of logs during failures." The window of potential loss is the buffer size (10 seconds), not minutes or hours.

Search API

GET /api/logs?service=payment-api&level=ERROR&from=2026-05-27T10:00:00Z&to=2026-05-27T11:00:00Z&q=timeout

Response:
{
  "total": 142,
  "logs": [
    { "timestamp": "...", "level": "ERROR", "message": "Failed to charge..." },
    ...
  ]
}

Query execution:

1. Determine which daily indices to search (from/to → logs-2026-05-27)
2. Filter by service and level (keyword match, fast)
3. Full-text search on message for "timeout" (inverted index)
4. Sort by timestamp descending
5. Return paginated results

Log Levels and Sampling

Not all logs are equal:

Sampling: At 50K logs/sec, storing everything for 30 days costs ~65 TB. To reduce costs:

• Keep 100% of ERROR/WARN
• Keep 100% of INFO for 14 days
• Sample DEBUG at 10% (store 1 in 10) or keep only 3 days

This is a cost vs observability tradeoff. Discuss it in the interview.

Key Takeaways

• Message queues decouple producers from consumers

  Kafka between app servers and storage means apps never block on log writes. If storage is slow or down, logs buffer in Kafka. This is the fundamental pattern for write-heavy systems.

• Time-based partitioning enables efficient queries and cleanup

  One index per day means time-range queries only scan relevant partitions. Deleting old data is instant (drop the index). No expensive row-by-row scans or garbage collection.

• Elasticsearch is purpose-built for log search

  Inverted indexes for full-text search, keyword fields for filtering, bulk API for high-throughput writes, horizontal sharding for scale. PostgreSQL can't match this at 50K writes/sec with full-text search.

• Graceful degradation preserves critical data

  When the system is overloaded, drop low-value logs (DEBUG) first. Keep ERROR logs at all costs. Design explicit priority levels so the system knows what to sacrifice under pressure.

• Bulk writes dramatically improve throughput

  Writing one log at a time means 50K round trips/sec. Batching 1,000 logs per request means 50 round trips/sec with the same data. Always batch writes to append-heavy stores.

• Cost vs observability is a real tradeoff

  Storing every DEBUG log for 30 days is expensive. Sampling, shorter retention for low-severity logs, and tiered storage (hot → cold → archive) are standard cost controls. State this tradeoff explicitly in interviews.

🎁 Storing scores is trivial. But can you compute a player's rank among 10 million others in under 50 milliseconds?