URL Shortener at Scale (100M URLs/day)

A URL shortener converts long URLs into short, shareable links (e.g., https://short.ly/abc123) and redirects users to the original URL when visited.

Step 1: URL Shortener Requirements

URL Shortener Functional Requirements

• Create short URL: given a long URL, generate a unique short link
• Redirect: visiting the short URL redirects to the original
• Custom aliases: users can optionally choose their own short code
• Expiration: URLs can have a TTL; expired links return 404
• Analytics: track click count, referrer, geo (nice-to-have)

URL Shortener Non-Functional Requirements

Why these matter: The 10:1 read-heavy ratio tells us to optimize reads (caching). The latency requirement means we need in-memory lookups for hot URLs.

Step 2: URL Shortener Estimation

Write QPS

100M URLs/day ÷ 86,400 sec/day ≈ 1,160 writes/sec
Peak (2x average): ~2,300 writes/sec

Using 1 day ≈ 100K seconds (easier mental math, ~16% safety margin):

100M ÷ 100K = 1,000 writes/sec
Peak (2x): ~2,000 writes/sec

Read QPS

10:1 ratio → 1B reads/day
1B ÷ 86,400 ≈ 11,600 reads/sec
Peak: ~23,000 reads/sec

With the 100K shortcut:

1B ÷ 100K = 10,000 reads/sec
Peak: ~20,000 reads/sec

URL Shortener Storage

Per URL record: ~500 bytes
  - short_code: 7 bytes
  - long_url: 200 bytes (avg)
  - created_at: 8 bytes
  - expires_at: 8 bytes
  - user_id: 16 bytes
  - metadata/overhead: ~260 bytes

5-year storage:
100M/day × 365 × 5 = 182.5B URLs
182.5B × 500 bytes = ~91 TB

Why 7 characters is enough: 26 uppercase + 26 lowercase + 10 digits = 62 possible characters. 62^7 = 3.5 trillion combinations. At 100M new URLs/day, that's enough for ~96 years.

URL Shortener Bandwidth

Write: 1,160 req/sec × 500 bytes = 580 KB/s (negligible)
Read:  11,600 req/sec × 500 bytes = 5.8 MB/s

Short Code Length

Base62 characters: [a-z, A-Z, 0-9] = 62 chars
62^7 = 3.5 trillion combinations → enough for 182.5B URLs
→ Use 7-character codes

Step 3: URL Shortener High-Level Design

                         ┌──────────────────┐
                         │  Load Balancer   │
                         └────────┬─────────┘
                                  │
                    ┌─────────────┼─────────────┐
                    │             │             │
              ┌─────▼────┐  ┌─────▼────┐  ┌─────▼────┐
              │ API Srv 1│  │ API Srv 2│  │ API Srv N│
              └─────┬────┘  └──────┬───┘  └─────┬────┘
                    │              │            │
                    └──────────────┼────────────┘
                                   │
                    ┌──────────────┼────────────┐
                    │                           │
              ┌─────▼──────┐              ┌─────▼──────┐
              │   Redis    │              │  Database  │
              │  (Cache)   │              │  (NoSQL)   │
              └────────────┘              └─────┬──────┘
                                                │
                                         ┌──────▼──────┐
                                         │  Replicas   │
                                         └─────────────┘

Component Roles

API Design

POST /api/shorten
  Body: { "url": "https://...", "custom_alias": "my-link", "expires_in": 86400 }
  Response: { "short_url": "https://short.ly/abc123" }

GET /:short_code
  Response: 302 redirect to original URL

Why 302, not 301? A 301 (Moved Permanently) tells the browser to cache the redirect, so future requests skip your server entirely. A 302 (Temporary Redirect) means every click hits your server first. For a URL shortener, 302 is the standard choice: you want to track clicks, enforce link expiration, and retain the ability to change the destination.

Step 4: URL Shortener Deep Dive

Short Code Generation

Three approaches compared:

Recommended: Counter + Base62 with distributed ID generator

Use a distributed unique ID generator (like Twitter Snowflake) to produce unique 64-bit IDs, then base62-encode them. This gives:

• No collisions (IDs are unique by construction)
• No coordination between servers (each gets an ID range)
• Non-sequential output (Snowflake includes timestamp + worker bits)

ID: 7294815263 → base62 → "kF3x9Q"

Twitter Snowflake generates 64-bit IDs by combining a 41-bit timestamp, 10-bit worker ID, and 12-bit sequence number. It's a lightweight function that runs locally on each server, no network calls, no coordination. Each worker can produce ~4 million unique IDs/sec.

Database Choice

Why NoSQL (DynamoDB / Cassandra) fits:

1. Simple access pattern: single key lookup (short_code → long_url). No joins, no complex queries.
2. Write-heavy at scale: NoSQL handles 2K+ writes/sec natively with horizontal partitioning.
3. Easy to shard: partition by short_code; even distribution with base62 codes.
4. Schema flexibility: easy to add fields (analytics, metadata) without migrations.

Schema:

Table: urls
  Partition Key: short_code (String)
  Attributes:
    - long_url (String)
    - created_at (Number)
    - expires_at (Number)
    - user_id (String)

If you need analytics queries (top URLs, user history), use a relational DB or add a secondary index.

Caching Strategy

Why cache: 20% of URLs generate 80% of traffic (Pareto principle). Caching hot URLs in Redis eliminates DB reads for the majority of requests.

Read path:
  1. Check Redis → HIT → redirect (< 1ms)
  2. MISS → query DB → store in Redis → redirect

Write path:
  1. Write to DB
  2. Optionally warm cache (write-through for popular creators)

Configuration:

• Eviction: LRU (Least Recently Used), which naturally keeps hot URLs
• TTL: Match URL expiration, or 24h for non-expiring URLs
• Size: ~20% of total URLs in cache covers 80% of reads
  • 182.5B × 20% × 500 bytes = ~18 TB (use Redis cluster)
  • Realistically, cache the last 30 days of hot URLs: much smaller

Handling Collisions

If using hash-based generation:

1. Generate hash of long_url
2. Take first 7 chars (base62)
3. Check DB for existing short_code
4. If collision → append counter and re-hash, or take next 7 chars
5. Retry up to 3 times, then fall back to counter-based generation

With counter-based approach: collisions are impossible since each ID is unique.

301 vs 302 Redirects

Decision: Use 302 by default. Why:

• Enables click tracking and analytics
• Allows changing the destination URL later
• Supports expiration (301 cached in browser ignores server-side expiry)
• The latency cost is minimal with Redis caching

Expiration and Cleanup

Approach: Lazy deletion + Background cleanup

Lazy (on read):
  1. User visits short URL
  2. Check expires_at
  3. If expired → return 404, delete from cache

Background (periodic):
  1. Cron job scans for expired URLs (expires_at < now)
  2. Batch delete from DB
  3. Invalidate cache entries
  4. Run during off-peak hours

Why both: Lazy deletion handles immediate correctness. Background cleanup reclaims storage and keeps the DB lean.

Analytics (Async via Message Queue)

┌──────────┐     ┌──────────┐     ┌──────────────┐     ┌───────────┐
│ API Srv  │────▶│  Kafka   │────▶│  Analytics   │────▶│ Analytics │
│(redirect)│     │  Queue   │     │  Consumer    │     │    DB     │
└──────────┘     └──────────┘     └──────────────┘     └───────────┘

Why async: Analytics must not add latency to redirects. If you write to an analytics database during the redirect request, the user waits for both the URL lookup AND the analytics write. Instead, fire-and-forget a message to Kafka on each redirect. The API server publishes the event and immediately returns the 302 response without waiting for acknowledgment:

{
  "short_code": "abc123",
  "timestamp": "2026-05-19T12:00:00Z",
  "ip": "203.0.113.1",
  "user_agent": "Mozilla/5.0...",
  "referrer": "https://twitter.com"
}

A separate analytics consumer reads messages from Kafka at its own pace, aggregates them (total clicks, clicks by country, by referrer, by hour), and writes to an analytics-optimized store (ClickHouse, TimescaleDB) for dashboards.

Why Kafka: It handles 23K+ events/sec (your peak read QPS) easily. If the consumer goes down, messages queue up and get processed when it recovers, so there's no data loss. You can also add more consumers later (fraud detection, A/B testing) without changing the redirect path.

The key insight: the redirect path and the analytics path are fully decoupled. The user never waits for analytics.

Key Takeaways

• Base62 encoding of unique IDs eliminates collisions and coordination overhead. Prefer it over hash-based approaches at scale.
• NoSQL is a natural fit when your access pattern is a single key-value lookup with no joins or complex queries.
• Cache the hot 20% with Redis + LRU eviction to serve 80% of reads in under 1ms. This is how you hit the <100ms latency target.
• Use 302 redirects unless you explicitly don't need analytics. The flexibility to track, expire, and update URLs outweighs the minor performance gain of 301.
• Decouple analytics with a message queue so tracking never blocks the redirect path. Availability and latency stay unaffected.
• Lazy + background deletion gives you both correctness (expired URLs immediately stop working) and efficiency (storage is reclaimed in bulk during off-peak).