RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #46 // Medium
#46MediumMonitoring, Telemetry & Logging

Design an On-Call Escalation System

ASKED IN INTERVIEWS AT:Uber

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design an enterprise incident alert escalation system (like PagerDuty) executing escalation policies, initiating multi-channel notifications, and tracking acknowledgments.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS100,000 engineers on call across 5,000 teams
READ / WRITE TRAFFIC RATIOBursty write operations during incidents
5-YEAR STORAGE CAPACITYTransactional PostgreSQL store with multi-region standby

FUNCTIONAL REQUIREMENTS

  • [1]Trigger incidents from monitoring alerts
  • [2]Page on-call engineer according to rotation schedules
  • [3]Escalate to manager if unacknowledged within 15 minutes

NON-FUNCTIONAL REQUIREMENTS

  • •100% alert delivery reliability (zero dropped pages)
  • •High availability during major cloud outages
  • •Audit log of every page and ack

CORE ARCHITECTURAL COMPONENTS

Alert Ingestion Webhook
Escalation Policy Engine
Schedule Rotation Resolver
Telephony Dispatcher (Twilio API)
Ack Listener Gateway

DATA SCHEMA & PERSISTENCE

Incident: id, service_id, status (TRIGGERED, ACKED, RESOLVED), current_level, acked_by

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Multi-provider telephony fallback (Twilio + AWS SNS) ensures alerts get through even when primary communications providers experience outages.