RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #39 // Medium
#39MediumMonitoring, Telemetry & Logging
Design a System to Monitor the Health of a Cluster
ASKED IN INTERVIEWS AT:UberLaceworkAmazonGoogle
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Design an infrastructure health checker detecting machine failures, split-brain states, and zombie processes across 50,000 servers using gossip protocols and heartbeats.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS50,000 cluster nodes monitored continuously
READ / WRITE TRAFFIC RATIOContinuous peer gossip and heartbeat checks
5-YEAR STORAGE CAPACITYIn-memory health state machine in etcd/Consul
FUNCTIONAL REQUIREMENTS
- [1]Detect node crashes, network partitions, and degraded disks
- [2]Notify scheduler to evacuate workloads from failing nodes
- [3]Provide real-time cluster health dashboard
NON-FUNCTIONAL REQUIREMENTS
- •Zero false positive evictions during transient network blips
- •Sub-5 second failure detection time
- •Low network overhead on cluster
CORE ARCHITECTURAL COMPONENTS
Gossip Protocol Daemon (SWIM)
Phi Accrual Suspicion Engine
Consul / etcd State Store
Cluster Node Evacuation Controller
Alerting Gateway
DATA SCHEMA & PERSISTENCE
NodeState: node_id, status (HEALTHY, SUSPECT, DEAD), last_ping, phi_score
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Phi Accrual failure detection calculates probabilistic failure based on heartbeat history instead of a rigid binary timeout.