08 - Design a Job Scheduler

🎁 "Run this at 3am every Tuesday" sounds simple until you have a million jobs, some fail halfway through, and your scheduler itself might crash.

A job scheduler runs tasks at specified times or intervals, think cron jobs at scale. Send emails at 9 AM, generate reports nightly, retry failed payments every 5 minutes. The challenge is reliability: jobs must run exactly once, even when servers crash, and the system must handle thousands of scheduled jobs without missing any.

Job Scheduler Requirements

Job Scheduler Functional Requirements

• Schedule a job to run at a specific time or on a recurring schedule (cron expression)
• Execute jobs reliably: every scheduled job runs exactly once
• Retry failed jobs with exponential backoff
• Support job priorities (urgent jobs run before low-priority ones)
• Track job status (pending, running, succeeded, failed, dead)

Job Scheduler Non-Functional Requirements

Job Scheduler Scope Exclusions

• DAG dependencies (job A must finish before job B starts)
• Multi-tenant isolation (separate job queues per customer)
• Job result storage (just track success/failure, not output data)

Job Scheduler Estimation

• 10M active jobs, 10K executions/minute peak → ~170 jobs/sec
• Job metadata: 10M × 1 KB = 10 GB (fits in one PostgreSQL instance)
• Queue throughput: 170 jobs/sec is trivial for Redis or SQS

Moderate scale. The challenge is reliability (exactly-once), not throughput.

Job Scheduler High-Level Design

┌────────────────────────────────────────────────────────────────┐
│                       JOB SCHEDULER                            │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌──────────┐     ┌──────────────┐     ┌──────────────────┐    │
│  │  Client  │────►│  API Server  │────►│  Job Store       │    │
│  │  (create │     │  (CRUD jobs) │     │  (PostgreSQL)    │    │
│  │   jobs)  │     └──────────────┘     │  schedule, status│    │
│  └──────────┘                          └────────┬─────────┘    │
│                                                 │              │
│                                                 ▼              │
│                                        ┌───────────────────┐   │
│                                        │  Scheduler        │   │
│                                        │  (polls for due   │   │
│                                        │   jobs, enqueues) │   │
│                                        └────────┬──────────┘   │
│                                                 │              │
│                                                 ▼              │
│                                        ┌──────────────────┐    │
│                                        │  Job Queue       │    │
│                                        │  (Redis or SQS, Amazon's managed queue service)      │    │
│                                        └────────┬─────────┘    │
│                                                 │              │
│                              ┌──────────────────┼──────────┐   │
│                              ▼                  ▼          ▼   │
│                        ┌──────────┐       ┌──────────┐         │
│                        │ Worker 1 │       │ Worker N │  ...    │
│                        │ (execute)│       │ (execute)│         │
│                        └──────────┘       └──────────┘         │
└────────────────────────────────────────────────────────────────┘

Flow:

1. Client creates a job (schedule, payload, priority)
2. Scheduler polls the Job Store every second for due jobs (WHERE run_at <= NOW())
3. Due jobs are pushed to the Job Queue
4. Workers pull from the queue and execute
5. On success: mark complete. On failure: retry with backoff.

Job Scheduler Deep Dive

Exactly-Once Execution

The hardest problem: ensuring a job runs exactly once, not zero times, not twice.

What can go wrong:

• Two schedulers poll at the same time → both enqueue the same job → runs twice
• Worker crashes mid-execution → job never completes → runs zero times

Solution: Claim with a lease

-- Scheduler claims due jobs atomically
UPDATE jobs
SET status = 'QUEUED', locked_until = NOW() + INTERVAL '5 minutes'
WHERE status = 'PENDING' AND run_at <= NOW() AND locked_until < NOW()
LIMIT 100
RETURNING *;

• locked_until prevents other schedulers from claiming the same job
• If the worker crashes, the lock expires and the job becomes claimable again
• Workers extend the lock periodically (heartbeat) while executing long jobs

Worker Failure Recovery

Worker picks up job → starts executing
  → Success: mark job as SUCCEEDED
  → Failure: increment retry_count, set run_at = NOW() + backoff, status = PENDING
  → Crash (no response): lock expires → scheduler re-enqueues it

Backoff: 5s, 30s, 2min, 10min, 1hr (exponential)
Max retries: 5 → then status = DEAD (alert the team)

Dead letter: Jobs that fail all retries move to DEAD status. An alert fires. Engineers investigate manually. Never auto-retry dead jobs. They failed for a reason.

Recurring Jobs

For jobs that repeat (e.g., "every day at 9 AM"):

Job: { id: "daily-report", cron: "0 9 * * *", next_run: "2026-05-28 09:00:00" }

After execution:
  1. Mark current run as SUCCEEDED
  2. Calculate next_run from cron expression
  3. Insert new job row with next_run (or update the same row)

Why not just use system cron? System cron runs on one machine. If that machine dies, jobs stop. A distributed job scheduler survives node failures because the Job Store is the source of truth, not any single server.

Job Priority Queue

Not all jobs are equal:

Queue structure:
  HIGH:   [payment-retry, alert-send]     ← processed first
  MEDIUM: [email-digest, report-gen]      ← processed after HIGH is empty
  LOW:    [cleanup, analytics-export]     ← processed last

Workers always pull from the highest-priority non-empty queue. A flood of low-priority analytics jobs never delays a payment retry.

Job Scheduler Scaling

Key Takeaways

• Lease-based claiming prevents double execution

  UPDATE WHERE locked_until < NOW() atomically claims jobs. If a worker dies, the lock expires and the job is retried. No distributed locks needed, just a timestamp column.

• Separate scheduling from execution

  The scheduler decides WHEN to run. Workers decide HOW to run. This separation lets you scale workers independently and swap execution logic without touching scheduling.

• Exponential backoff with a dead letter queue

  Retry with increasing delays (5s → 30s → 2min → 10min). After max retries, stop and alert. Never retry forever. It wastes resources and hides bugs.

• Priority queues prevent starvation of critical jobs

  A flood of low-priority work should never delay high-priority jobs. Separate queues per priority with workers draining high first.

• Distributed beats single-machine cron

  System cron is a single point of failure. A job scheduler with a durable store survives node failures. Any scheduler instance can pick up due jobs.

Want the full design? The example "Job Scheduler at Scale" covers DAG dependencies, leader election, partitioned scheduling, and FOR UPDATE SKIP LOCKED.

🎁 One event triggers a push notification, an email, and an SMS, each through a different provider with different failure modes. How do you guarantee delivery?