File Storage Service at Scale

📋 Jump to Takeaways

Step 1: File Storage Requirements

File Storage Functional Requirements

Upload/download files: support files up to 10GB
File versioning: maintain history of changes, allow rollback
Sharing: shareable links (public/private), granular permissions (view, edit, owner)
Sync across devices: changes on one device reflect on all others in near real-time
Folder structure: hierarchical organization with nested directories
Offline support: clients work offline and sync when reconnected

File Storage Non-Functional Requirements

500M total users, 100M DAU
99.99% durability: never lose a file once uploaded
99.9% availability for reads, 99.95% for writes
Average file size: 500KB
2 files uploaded per user per day
Sync latency < 5 seconds for online devices

Step 2: File Storage Estimation

File Storage Storage

Uploads/day: 100M DAU × 2 files = 200M files/day
Storage/day: 200M × 500KB = 100TB/day
Storage/year: 100TB × 365 = ~36.5PB/year

File Storage Bandwidth

Upload: 100TB/day = ~1.2GB/s sustained
Download (assume 3:1 read-to-write): ~3.6GB/s sustained
Peak (5× average): ~18GB/s download bandwidth

Metadata

Each file: ~500 bytes of metadata (name, path, owner, timestamps, permissions, block list)
200M files/day × 500B = 100GB metadata/day
After 3 years: 100B files × 500B = **50TB metadata**

QPS

Upload QPS: 200M / 86400 = ~2,300 uploads/sec (peak: ~12K/sec)
Download QPS: ~7,000/sec (peak: ~35K/sec)

Step 3: File Storage High-Level Design

┌──────────┐         ┌──────────────┐        ┌──────────────────┐
│  Client  │────────▶│  API Gateway │───────▶│  Upload Service  │
│  (App)   │◀────────│  (Auth/Rate) │        └────────┬─────────┘
└────┬─────┘         └──────┬───────┘                 │
     │                      │                         ▼
     │                      │               ┌──────────────────┐
     │                      │               │  Block Storage   │
     │                      │               │  (S3 / GCS)      │
     │                      ▼               └──────────────────┘
     │              ┌───────────────┐
     │              │ Metadata DB   │ (PostgreSQL, sharded)
     │              │ (files, blocks│
     │              │  versions)    │
     │              └───────────────┘
     │
     │    ┌─────────────────┐       ┌──────────────────┐
     └───▶│  Sync Service   │◀─────▶│ Notification Svc │
          │ (change log)    │       │ (WebSocket/SSE)  │
          └─────────────────┘       └──────────────────┘
                                              │
                                    ┌─────────▼─────────┐
                                    │  Message Queue    │
                                    │  (Kafka/SQS)      │
                                    └───────────────────┘

Downloads: Client → CDN (CloudFront) → Block Storage (cache miss)

File Storage Component Responsibilities

API Gateway: authentication, rate limiting, routing
Upload Service: receives chunks, writes to block storage, updates metadata
Metadata DB: file tree, block references, versions, sharing permissions
Block Storage (S3): immutable file blocks, replicated across AZs
Sync Service: tracks change log per user, serves diffs to clients
Notification Service: pushes change events to connected devices via WebSocket
CDN: caches popular files close to users for fast downloads

Step 4: File Storage Deep Dive

Chunked Uploads

Large files are split into 4MB blocks on the client side:

Client splits file → computes SHA-256 hash per block
Client sends block list to server: [{hash, size, index}]
Server responds with which blocks are already stored (dedup check)
Client uploads only missing blocks in parallel
On failure, client resumes from the last unacknowledged block

Benefits: resumable uploads, parallelism, deduplication at block level.

File Storage Deduplication

Each block is identified by its content hash (SHA-256)
Before storing, check if hash already exists in block index
If it exists → just add a reference (no new storage)
Cross-user dedup: common files (OS installers, libraries) stored once
Saves 40-60% storage in practice (Dropbox reported similar numbers)

Metadata vs Data Separation

Concern	Storage	Why
File metadata	PostgreSQL (sharded by user_id)	ACID, complex queries, permissions
Block data	Object storage (S3)	Cheap, durable (11 nines), scalable
Block index	Key-value store (DynamoDB)	Fast hash lookups

Schema (simplified):

files: id, user_id, name, parent_folder_id, latest_version_id, created_at, updated_at
versions: id, file_id, block_list[], size, created_at, created_by
blocks: hash (PK), size, storage_path, ref_count
sharing: file_id, user_id, permission_level, share_token

Sync Protocol

Client maintains a local snapshot (file tree + block hashes)
On file change, client computes diff → uploads new blocks → updates metadata
Server appends to change log (per-user ordered event stream)
Other devices poll or receive push: "changes since cursor X"
Client applies remote changes to local state, downloads new blocks as needed

Change notification delivery:

WebSocket for active/online clients (low latency)
Long-polling as fallback for restricted networks
Push notification for mobile devices (wake app to sync)

Conflict Resolution

When two devices edit the same file offline:

Both upload their versions
Server detects conflict (parent version doesn't match)
Strategy: last-writer-wins for the primary copy
Losing edit saved as a conflict copy (e.g., report (Serjik's conflicted copy).docx)
User manually resolves by choosing which to keep

This matches Dropbox's approach: simple, predictable, no data loss.

Versioning

Each version stores a new block list, not a full copy
If only 2 of 100 blocks changed, version references 98 existing + 2 new blocks
Old versions can be garbage-collected after retention period (e.g., 30 days for free tier)
Delta encoding between versions for additional savings on text-heavy files

Storage Optimization

Compression: blocks compressed with LZ4 (fast) or Zstandard (better ratio) before storage
Tiered storage: hot (S3 Standard) → warm (S3-IA after 30 days) → cold (Glacier after 90 days)
Garbage collection: background job decrements ref_count when versions are deleted; blocks with ref_count=0 are purged
Encryption: blocks encrypted at rest (AES-256); per-user keys for enterprise tier

Key Takeaways

Chunking is fundamental: it enables resumable uploads, parallel transfers, deduplication, and efficient versioning all at once.
Separate metadata from data: SQL for structure/permissions, object storage for raw bytes. They scale differently and have different consistency needs.
Content-addressable storage (hash-based) gives you deduplication for free and makes integrity verification trivial.
Sync is the hardest problem: maintain an ordered change log per user, use cursors for incremental sync, and push notifications for low-latency updates.
Conflict resolution must never lose data: last-writer-wins with conflict copies is simple and battle-tested (Dropbox, Google Drive both use variants of this).
Storage costs dominate at scale: tiered storage, compression, and dedup are not optimizations but requirements at petabyte scale.