System Design Classic › 10. Design a News Feed › Example

News Feed at Scale

📋 Jump to Takeaways

A news feed is the personalized, algorithmically-ranked stream of posts you see on platforms like Facebook, Twitter/X, Instagram, or LinkedIn. It aggregates content from everyone you follow, ranks it by relevance, and delivers it in near real-time. This is different from an RSS feed (a pull-based XML format for subscribing to websites). A news feed is centralized, push-based, and the platform controls what you see and in what order.

The system design challenge: how do you build this at 500M users, where a single celebrity post must reach 10M followers instantly?

Step 1: News Feed Requirements

News Feed Functional Requirements

  • Users create posts (text, images, videos, links)
  • Users follow/unfollow other users
  • Users see a personalized home feed of posts from people they follow
  • Like and comment on posts
  • Media attachments (images, short videos, link previews)
  • Feed supports infinite scroll with pagination

News Feed Non-Functional Requirements

  • 300M daily active users (DAU)
  • Feed loads in < 500ms (p99)
  • Eventual consistency is acceptable (new post appears within 5 seconds)
  • Users can follow up to 5,000 people
  • Support celebrities with 10M+ followers
  • High availability (99.99% uptime)
  • Feed should feel "fresh", no stale content older than a few hours at the top

Step 2: News Feed Estimation

News Feed Traffic

  • 300M DAU, each user checks feed ~10x/day → 3B feed reads/day → ~35K reads/sec
  • 10% of users post daily → 30M posts/day → ~350 posts/sec
  • Average user follows 200 people

Fanout Volume

  • Each post fans out to average 200 followers → 30M × 200 = 6B fanout writes/day
  • Celebrity post (10M followers) would require 10M writes, too expensive on write path

News Feed Storage

  • Post metadata: ~1KB per post → 30M posts/day × 1KB = 30GB/day → ~11TB/year
  • Feed cache entry: post ID (8 bytes) + score (8 bytes) = 16 bytes per entry
  • Feed cache per user: 500 posts × 16 bytes = 8KB
  • Hot users in cache (100M users): 100M × 8KB = 800GB Redis

Media

  • 20% of posts have images (~2MB avg): 6M × 2MB = 12TB/day → CDN + object storage

Step 3: News Feed High-Level Design

┌────────┐       ┌──────────────┐       ┌────────────────┐
│ Client │──────▶│  API Gateway │──────▶│  Post Service  │
└────────┘       └──────────────┘       └───────┬────────┘
     │                                          │
     │                                          ▼
     │                                  ┌────────────────┐
     │                                  │ Fanout Service │
     │                                  └───────┬────────┘
     │                                          │
     │                                          ▼
     │           ┌──────────────┐       ┌───────────────┐
     │◀──────────│ Feed Service │◀──────│  Feed Cache   │
     │           └──────────────┘       │   (Redis)     │
     │                                  └───────────────┘
     │
     │           ┌──────────────┐       ┌───────────────┐
     └──────────▶│ User Service │──────▶│  User Graph   │
                 └──────────────┘       │  (who follows │
                                        │    whom)      │
                                        └───────────────┘

Supporting services:
┌───────────────┐  ┌────────────────────┐  ┌──────────────┐
│ Media Service │  │ Notification Svc   │  │ Post Storage │
│ (upload/CDN)  │  │ (push on mention)  │  │ (PostgreSQL) │
└───────────────┘  └────────────────────┘  └──────────────┘

Flow: Publishing a Post

  1. Client sends post to Post Service
  2. Post Service writes to Post Storage (source of truth)
  3. Post Service emits event to Fanout Service (async via message queue)
  4. Fanout Service looks up followers in User Graph
  5. For normal users (< 500K followers): push post ID into each follower's Feed Cache
  6. For celebrities: skip fanout (handled at read time)

Flow: Reading the Feed

  1. Client requests feed from Feed Service
  2. Feed Service reads user's pre-built feed from Feed Cache (Redis sorted set)
  3. Feed Service merges in recent posts from followed celebrities (fan-out on read)
  4. Hydrates post IDs → full post objects from Post Storage
  5. Returns ranked, paginated feed to client

Step 4: News Feed Deep Dive

Fan-Out on Write vs Fan-Out on Read

Approach Pros Cons
Fan-out on write Fast reads, pre-computed feed Expensive for celebrities, wasted work for inactive users
Fan-out on read No write amplification Slow reads, must merge N sources

Hybrid approach (what Facebook/Twitter use):

  • Normal users (< 500K followers): fan-out on write, push post ID to all followers' caches
  • Celebrities (> 500K followers): fan-out on read, merge at query time
  • Threshold is tunable; start at 500K, adjust based on latency metrics

Feed Generation and Ranking

Simple chronological feed:

  • Redis sorted set, score = Unix timestamp
  • Newest posts first, trivial to implement

Ranked feed (production systems):

  • Score = base_timestamp + engagement_boost + affinity_score
  • Affinity: how often you interact with this person
  • Recency: exponential decay over time
  • Engagement: likes/comments in first N minutes boost visibility

For an interview, describe chronological first, then mention ranking as an enhancement.

Feed Cache Structure

Key:   feed:{user_id}
Type:  Redis Sorted Set
Score: Unix timestamp (ms)
Value: post_id

Example:
  ZADD feed:user123 1716000000000 "post_abc"
  ZADD feed:user123 1716000060000 "post_def"

Read feed (newest 20):
  ZREVRANGEBYSCORE feed:user123 +inf -inf LIMIT 0 20
  • Keep only the latest 500–1000 entries per user (ZREMRANGEBYRANK to trim)
  • TTL of 7 days on inactive feeds. Rebuild from Post Storage on cache miss
  • Total cache: ~800GB fits in a Redis cluster (10–20 nodes)

The Celebrity Problem

A user with 10M followers posting once means 10M cache writes. At 350 posts/sec from celebrities, this creates write storms.

Solution:

  1. Maintain a "celebrity list" (users with > 500K followers)
  2. When a celebrity posts, do NOT fan out. Just store the post
  3. When a follower reads their feed:
    • Fetch pre-built feed from cache (posts from normal followees)
    • Fetch recent posts from each followed celebrity (small list, typically < 50)
    • Merge and rank both sets
  4. Cache the merged result briefly (30–60s TTL) to avoid repeated celebrity fetches

Post Storage

Feed cache stores only post IDs, not full post content.

Posts table (PostgreSQL / sharded):
  post_id     BIGINT PRIMARY KEY
  author_id   BIGINT
  content     TEXT
  media_urls  JSONB
  created_at  TIMESTAMP
  like_count  INT
  comment_count INT
  • Shard by post_id (snowflake ID embeds timestamp)
  • Read-through cache (Redis/Memcached) for hot posts
  • Separating post storage from feed means updating a post (edit, delete) doesn't require touching millions of feed caches

Pagination

Cursor-based pagination using the timestamp score:

GET /feed?cursor=1716000060000&limit=20

→ ZREVRANGEBYSCORE feed:{user_id} (1716000060000 -inf LIMIT 0 20
  • Client passes the score of the last seen post as cursor
  • Avoids offset-based issues (new posts shifting pages)
  • Handles real-time insertions gracefully, no duplicates, no skips

Cache Invalidation

  • New post from normal user: Fanout Service pushes post ID into each follower's sorted set
  • Deleted post: Remove post ID from author's followers' caches (async, best-effort) + remove from Post Storage
  • Unfollow: Lazy cleanup. Stale posts from unfollowed user get filtered at read time, eventually evicted by trim
  • Cache miss: Rebuild feed from Post Storage by querying recent posts from all followed users (expensive but rare)

Media Delivery and CDN Sizing

20% of posts have images (~2MB average), generating 12TB/day of media uploads. This media must be served efficiently at read time.

CDN strategy:

  • Media uploaded to object storage (S3) via pre-signed URLs
  • CDN pulls from origin on first request, caches at edge
  • Hot media (< 24h old, viral posts) stays cached; long-tail evicted by LRU
  • Geographic routing via DNS/anycast to nearest edge PoP

Bandwidth estimation:

3B feed reads/day × 5 media items per feed page × 30% cache-miss rate
= ~4.5B media requests/day from origin
At 200KB average (resized thumbnails): 4.5B × 200KB = 900TB/day from CDN edge

CDN edge bandwidth: 900TB / 86,400s ≈ 10.4 TB/s (83 Tbps)

This is served across hundreds of CDN PoPs globally, each handling a fraction. With a 70%+ cache hit rate at edge, origin bandwidth drops to ~25 Tbps.

Cost implication: At ~$0.02/GB for CDN egress, 900TB/day = ~$18K/day ($540K/month). This is why feed systems aggressively resize images (serve thumbnails in feed, full-res on tap) and use modern formats (WebP/AVIF for 30-50% size reduction).

Key Takeaways

  • Use a hybrid fanout strategy: fan-out on write for normal users, fan-out on read for celebrities. This balances write cost against read latency.
  • Redis sorted sets are ideal for feed caches: O(log N) insert, O(log N + M) range queries, natural ordering by timestamp.
  • Store only post IDs in the feed cache: hydrate full post objects separately. This keeps cache small and decouples post updates from feed updates.
  • Cursor-based pagination with timestamps avoids the duplicate/skip problems of offset pagination in real-time feeds.
  • The celebrity problem is the core design challenge: recognizing and solving it demonstrates you understand write amplification (one post causing millions of cache writes) tradeoffs.
  • Eventual consistency is acceptable: users won't notice a 2-5 second delay for new posts appearing in followers' feeds, and this unlocks async processing via message queues.
© 2026 ByteLearn.dev. Free courses for developers. · Privacy
☕