10 - Design a News Feed
📋 Jump to Takeaways🎁 A celebrity with 50 million followers posts an update. Does the system pre-compute 50 million feeds, or does each follower assemble their own on the fly?
A news feed is the personalized stream of posts you see on Facebook, Twitter/X, or LinkedIn. It aggregates content from everyone you follow, ranks it, and delivers it in near real-time. The core challenge: when a user with 10 million followers posts, how do you get that post into 10 million feeds quickly?
News Feed Requirements
News Feed Functional Requirements
- Users create posts (text, images, links)
- Users follow/unfollow other users
- Home feed shows posts from followed users, ranked by relevance
- Feed supports infinite scroll with pagination
- New posts appear in followers' feeds within seconds
News Feed Non-Functional Requirements
| Metric | Target |
|---|---|
| DAU | 500 million |
| Posts/day | 100 million new posts |
| Feed reads/day | 5 billion (users refresh frequently) |
| Feed latency | < 200ms to load |
| Freshness | New posts visible within 5 seconds |
News Feed Scope Exclusions
- Post creation and media upload (separate service)
- Comments and likes (separate engagement service)
- Ad insertion into the feed
News Feed Estimation
- 500M DAU × 10 feed reads/day = 5B reads/day → ~58K reads/sec
- 100M new posts/day → ~1,160 writes/sec
- Feed cache: 500M users × 500 post IDs × 8 bytes = ~2 TB Redis
Read-heavy at massive scale. Caching is essential. You cannot query the database 58K times/sec.
News Feed High-Level Design
┌────────────────────────────────────────────────────────────────┐
│ NEWS FEED │
├────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ User │────►│ Post │────►│ Post Store │ │
│ │ (write) │ │ Service │ │ (database) │ │
│ └──────────┘ └──────┬───────┘ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ Fan-out │ │
│ │ Service │ │
│ └──────┬───────┘ │
│ │ push post_id to each │
│ │ follower's feed cache │
│ ▼ │
│ ┌──────────────┐ │
│ │ Feed Cache │ │
│ │ (Redis │ │
│ │ sorted set │ │
│ │ per user) │ │
│ └──────┬───────┘ │
│ │ │
│ ┌──────────┐ │ │
│ │ User │◄───────────┘ │
│ │ (read) │ GET feed → sorted post_ids → hydrate content │
│ └──────────┘ │
└────────────────────────────────────────────────────────────────┘News Feed Deep Dive
Fan-out on Write vs Read
Fan-out means distributing data to multiple destinations, like dealing cards to players.
| Approach | How it works | Pros | Cons |
|---|---|---|---|
| Fan-out on write | When user posts, push to all followers' caches immediately | Fast reads (feed is pre-built) | Expensive writes for celebrities (10M pushes) |
| Fan-out on read | When user opens feed, fetch posts from all followed users on demand | Cheap writes | Slow reads (must query many users) |
Hybrid approach (what Facebook/Twitter use):
- Normal users (< 10K followers): fan-out on write
- Celebrities (> 10K followers): fan-out on read (merge at read time)
This avoids the "celebrity problem." A celebrity posting shouldn't trigger 10M cache writes.
Feed Cache with Redis Sorted Sets
A Redis sorted set (covered in lesson 05: Leaderboard) maps members to scores and keeps them sorted. Here, member = post_id, score = timestamp. This gives us a pre-sorted feed per user.
Each user has a sorted set of post IDs, scored by timestamp:
feed:user_123 → { post_A: 1716800100, post_B: 1716800050, post_C: 1716799900 }Reading the feed:
ZREVRANGE feed:user_123 0 19→ top 20 post IDs- Batch-fetch post content from Post Store
- Return to client
Why only store post IDs, not full content? If a user edits their post, you'd need to update it in millions of feed caches. Storing IDs means the feed cache is tiny and content is fetched fresh.
Cursor-Based Pagination
New posts are constantly added. Offset-based pagination (page=2) breaks. Items shift and you see duplicates or miss posts.
Cursor-based: "Give me posts older than this timestamp"
GET /feed?cursor=1716800050&limit=20
→ Returns posts with timestamp < 1716800050
→ Response includes next_cursor for the next pageStable regardless of new posts being added above.
Feed Ranking Signals
A simple chronological feed (newest first) is the baseline. Production feeds add ranking signals:
- Recency: newer posts score higher (exponential decay)
- Affinity: posts from users you interact with frequently rank higher
- Engagement: posts with early likes/comments get boosted
For this design, chronological (sorted by timestamp) is sufficient. Ranking is an ML layer on top.
Key Takeaways
Hybrid fan-out solves the celebrity problem
Fan-out on write for normal users (fast reads). Fan-out on read for celebrities (avoid 10M writes per post). Merge both at read time.
Store only post IDs in the feed cache
Keeps the cache small and avoids stale content when posts are edited. Hydrate content separately on read.
Cursor-based pagination for constantly-changing feeds
Offset pagination breaks when new items are inserted. A cursor (last seen timestamp) gives a stable position regardless of new posts above.
Redis sorted sets are ideal for feed storage
Score = timestamp, member = post_id. ZREVRANGE gives top-N instantly. ZREMRANGEBYRANK trims old entries to bound cache size.
Want the full design? The example "News Feed at Scale" covers feed ranking algorithms, celebrity detection thresholds, cache warming strategies, and CDN integration.
🎁 Two people are typing messages to each other at the same time from different continents. How do you deliver each message in under 100ms?