RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #46 // Medium
#46MediumMonitoring, Telemetry & Logging
Design an On-Call Escalation System
ASKED IN INTERVIEWS AT:Uber
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Design an enterprise incident alert escalation system (like PagerDuty) executing escalation policies, initiating multi-channel notifications, and tracking acknowledgments.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS100,000 engineers on call across 5,000 teams
READ / WRITE TRAFFIC RATIOBursty write operations during incidents
5-YEAR STORAGE CAPACITYTransactional PostgreSQL store with multi-region standby
FUNCTIONAL REQUIREMENTS
- [1]Trigger incidents from monitoring alerts
- [2]Page on-call engineer according to rotation schedules
- [3]Escalate to manager if unacknowledged within 15 minutes
NON-FUNCTIONAL REQUIREMENTS
- •100% alert delivery reliability (zero dropped pages)
- •High availability during major cloud outages
- •Audit log of every page and ack
CORE ARCHITECTURAL COMPONENTS
Alert Ingestion Webhook
Escalation Policy Engine
Schedule Rotation Resolver
Telephony Dispatcher (Twilio API)
Ack Listener Gateway
DATA SCHEMA & PERSISTENCE
Incident: id, service_id, status (TRIGGERED, ACKED, RESOLVED), current_level, acked_by
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Multi-provider telephony fallback (Twilio + AWS SNS) ensures alerts get through even when primary communications providers experience outages.