#23EasyCore Distributed Systems

Design a Job Scheduler

ASKED IN INTERVIEWS AT:GoogleAmazonMicrosoftDoorDash

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design a distributed cron-like task execution engine that schedules, distributes, and executes millions of jobs reliably at exact timestamps with retries.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS10 Million scheduled jobs executed per day
READ / WRITE TRAFFIC RATIOBalanced write and execution state transitions
5-YEAR STORAGE CAPACITYRelational DB metadata store with partition per month

FUNCTIONAL REQUIREMENTS

  • [1]Schedule one-off and recurring jobs with cron expressions
  • [2]Execute jobs across worker fleet
  • [3]Retry failed tasks with exponential backoff

NON-FUNCTIONAL REQUIREMENTS

  • •Job execution delay < 1s
  • •Zero duplicate task executions
  • •Scales to millions of scheduled jobs

CORE ARCHITECTURAL COMPONENTS

Scheduler Leader (Timing Wheel)
Distributed Lock Manager
Worker Task Queue (RabbitMQ)
Execution Worker Pool
Audit Heartbeat Log

DATA SCHEMA & PERSISTENCE

ScheduledJobs: job_id, scheduled_at, cron_expr, payload, status (PENDING, RUNNING, COMPLETED, FAILED)

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Timing Wheel algorithm allows O(1) scheduling and execution lookup, vastly outperforming scanning database tables every second.