01 - Design a Pastebin

🎁 How do you store and retrieve millions of text snippets when each one needs its own unique URL that never collides?

A Pastebin (like pastebin.com or GitHub Gists) lets users paste text, get a unique URL, and share it. It's the simplest system design problem, essentially a key-value store with a web interface. This makes it a perfect starting point for learning the system design process.

Pastebin Requirements

Pastebin Functional Requirements

• User pastes text content and receives a unique short URL
• Anyone with the URL can view the paste
• Pastes can have an optional expiration time
• Syntax highlighting for code (client-side, not a backend concern)

Pastebin Non-Functional Requirements

Pastebin Scope Exclusions

• User accounts or authentication (anonymous pastes)
• Edit or versioning (pastes are immutable once created)
• Search (no full-text search across pastes)

Keeping scope small is critical in interviews. State what you're excluding and why.

Pastebin Estimation

Pastebin Traffic

• 1M DAU × 2 pastes/day = 2M writes/day
• 2M × 5 (read:write) = 10M reads/day
• Write QPS: 2M / 86,400 ≈ 23 writes/sec
• Read QPS: 10M / 86,400 ≈ 115 reads/sec

This is low traffic. A single server could handle it. But we design for growth and reliability.

Pastebin Storage

• Average paste: 10 KB (most pastes are small code snippets)
• Daily new storage: 2M × 10 KB = 20 GB/day
• Yearly: 20 GB × 365 = 7.3 TB/year
• 5-year retention: ~36 TB

Pastebin Bandwidth

• Write: 23/sec × 10 KB = 230 KB/sec (trivial)
• Read: 115/sec × 10 KB = 1.15 MB/sec (trivial)

Key insight: This system is storage-bound, not compute-bound. The challenge is storing and retrieving billions of pastes efficiently, not handling high QPS.

Pastebin High-Level Design

┌─────────────────────────────────────────────────────────────┐
│                        PASTEBIN                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────┐     ┌──────────────┐     ┌──────────────┐     │
│  │  Client  │────►│  API Server  │────►│  Metadata DB │     │
│  │          │◄────│              │     │  (PostgreSQL)│     │
│  └──────────┘     └──────┬───────┘     └───────┬──────┘     │
│                          │                     ▲            │
│                          ▼                     │            │
│                   ┌──────────────┐      ┌──────┴───────┐    │
│                   │  Object      │      │  Cleanup Job │    │
│                   │  Storage     │◄─────│  (cron)      │    │
│                   │  (S3)        │      │ finds expired│    │
│                   │ paste content│      │  deletes both│    │
│                   └──────────────┘      └──────────────┘    │
└─────────────────────────────────────────────────────────────┘

Why this split?

• Metadata DB (PostgreSQL): Stores paste_id, created_at, expires_at, size. Small rows, fast lookups by ID.
• Object Storage (S3): Stores the actual paste content. Cheap, durable, scales to petabytes.
• Cleanup Job: Queries Metadata DB for expired pastes, deletes from both S3 and PostgreSQL.

Pastebin API Design

POST /api/pastes
  Body: { content: "...", expires_in: 3600 }
  Response: 201 Created
    { id: "xK9mP2qL", url: "https://paste.example.com/xK9mP2qL" }
  Errors:
    400 - content empty or exceeds 10 MB
    429 - rate limited (too many pastes from this IP)

GET /api/pastes/:id
  Response: 200 OK
    { id: "xK9mP2qL", content: "...", created_at: "...", expires_at: "..." }
  Errors:
    404 - paste not found or expired

Pastebin Write Flow

What happens when a user creates a paste:

1. API Server generates a random base62 ID
2. Store content in S3 at key pastes/{id}
3. Write metadata to PostgreSQL (id, created_at, expires_at, s3_key)
4. Return the URL to the client

What if S3 succeeds but PostgreSQL fails? You have an orphaned blob in S3. Two options:

• Wrap in a transaction-like pattern: write metadata first with status PENDING, upload to S3, then update status to ACTIVE. Cleanup job deletes PENDING entries older than 5 minutes.
• Accept rare orphans. A periodic S3 reconciliation job finds blobs without matching metadata and deletes them.

Pastebin Deep Dive

Paste ID Generation

We need short, unique IDs for URLs. Options:

Best choice for Pastebin: Base62 random (8 chars)

8 characters of base62 = 62^8 ≈ 218 trillion combinations. At 2M pastes/day, collision probability is negligible. Generate randomly, check if exists (fast index lookup), retry on collision.

Generate: random 8-char base62 string → "xK9mP2qL"
URL: https://paste.example.com/xK9mP2qL

What about auto-increment + base62? This is a valid technique used by URL shorteners (bit.ly, t.co). A counter gives you 1, 2, 3... and you encode each to base62 (1→"1", 62→"10", 3844→"100"). Zero collisions, no retry logic needed. The downside: anyone who knows the scheme can decode "xK9mP2qL" back to a number, increment it, re-encode, and access the next paste. For a URL shortener (where links are public anyway) this is fine. For a Pastebin (where pastes may be private), it leaks content.

Why not hash the content? Two users pasting the same text should get different URLs. They may have different expiry times, and if one user deletes their paste, the other's should still work.

Metadata vs Content Separation

You might ask: "Why not store everything in PostgreSQL?"

The split is the standard pattern: relational DB for structured queries (find expired pastes, count pastes per day) + object storage for blobs (cheap, durable, no size limits).

Paste Expiration Strategies

Three approaches:

1. Lazy deletion: Check expires_at on every read. If expired, return 404 and delete. Simple but doesn't free storage until someone tries to access it.

2. Background cleanup job: Cron job runs every hour, queries WHERE expires_at < NOW(), deletes in batches. Frees storage proactively.

3. Database TTL: Some databases (DynamoDB, Cassandra) support native TTL. Items auto-delete. No cleanup job needed.

Best: Lazy deletion + background job. Lazy deletion gives immediate correctness (never serve expired content). Background job reclaims storage for pastes nobody ever accesses again.

Pastebin Caching Strategy

At 115 reads/sec we don't need caching. But if traffic grows 100×:

Client → CDN (cache popular pastes at edge)
       → API Server → Redis (cache recent/hot pastes)
                    → PostgreSQL + S3 (cache miss)

What do we cache? Metadata and content together as a single object, because every read needs both. There's no use case where you'd fetch metadata without content.

Redis key:   paste:xK9mP2qL
Redis value: { content: "...", created_at: "...", expires_at: "..." }
TTL:         matches paste expiry (or 1 hour for no-expiry pastes)

CDN vs Redis: which to use?

For a basic Pastebin where pastes are immutable, CDN is the better first choice. It's simpler, cheaper, and serves globally. Add Redis only if you need instant invalidation (e.g., user deletes a paste and it must disappear immediately).

Key Takeaways

• Separate metadata from content

  Structured data (IDs, timestamps, expiry) goes in a relational DB for queries. Blobs (paste content) go in object storage for cheap, durable, scalable storage. This pattern appears in almost every system design.

• Base62 random IDs are the default choice for short URLs

  8 characters give 218 trillion combinations. Generate randomly, check for collision, retry. Simple, fast, and URL-safe.

• Lazy deletion + background cleanup is the standard expiry pattern

  Check on read (never serve stale data) + periodic cleanup (reclaim storage). This combination gives both correctness and efficiency.

• Start simple, identify the real bottleneck

  Pastebin is storage-bound, not compute-bound. The estimation step told us this: 23 writes/sec is trivial, but 36 TB over 5 years needs planning. Always let the numbers guide your design focus.

• State what you're NOT building

  Excluding features (auth, search, editing) keeps the interview focused. It shows you can scope a problem and prioritize.

🎁 What happens when the system needs to sync state across devices, handle conflicts, and support offline edits? A to-do list is deceptively hard.