RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #39 // Medium
#39MediumMonitoring, Telemetry & Logging

Design a System to Monitor the Health of a Cluster

ASKED IN INTERVIEWS AT:UberLaceworkAmazonGoogle

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design an infrastructure health checker detecting machine failures, split-brain states, and zombie processes across 50,000 servers using gossip protocols and heartbeats.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS50,000 cluster nodes monitored continuously
READ / WRITE TRAFFIC RATIOContinuous peer gossip and heartbeat checks
5-YEAR STORAGE CAPACITYIn-memory health state machine in etcd/Consul

FUNCTIONAL REQUIREMENTS

  • [1]Detect node crashes, network partitions, and degraded disks
  • [2]Notify scheduler to evacuate workloads from failing nodes
  • [3]Provide real-time cluster health dashboard

NON-FUNCTIONAL REQUIREMENTS

  • •Zero false positive evictions during transient network blips
  • •Sub-5 second failure detection time
  • •Low network overhead on cluster

CORE ARCHITECTURAL COMPONENTS

Gossip Protocol Daemon (SWIM)
Phi Accrual Suspicion Engine
Consul / etcd State Store
Cluster Node Evacuation Controller
Alerting Gateway

DATA SCHEMA & PERSISTENCE

NodeState: node_id, status (HEALTHY, SUSPECT, DEAD), last_ping, phi_score

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Phi Accrual failure detection calculates probabilistic failure based on heartbeat history instead of a rigid binary timeout.