RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #17 // Hard
#17HardMonitoring, Telemetry & Logging
System to Collect Performance Metrics from Thousands of Servers
ASKED IN INTERVIEWS AT:GoogleDatadogAmazoneBay
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Build a telemetry collection architecture aggregating system utilization metrics (CPU, disk, network, JVM) from 100,000+ nodes without overwhelming target networks.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS100,000 host nodes * 50 metrics * every 5s = 1 Million metrics/sec
READ / WRITE TRAFFIC RATIOPure continuous write stream
5-YEAR STORAGE CAPACITYCompressed protocol buffer payloads: ~50 MB/sec ingestion
FUNCTIONAL REQUIREMENTS
- [1]Lightweight agent collects OS metrics every 5 seconds
- [2]Buffer and stream metrics to central aggregation cluster
- [3]Detect dead or failing hosts
NON-FUNCTIONAL REQUIREMENTS
- •Agent CPU overhead < 1%
- •Resilient to network partitions (local agent spooling)
- •Support 100K+ concurrent agents
CORE ARCHITECTURAL COMPONENTS
Local Host Daemon (eBPF / Go agent)
Load Balancer Layer
Kafka Stream Buffer
Aggregator Workers
Time-Series Long Term Store
DATA SCHEMA & PERSISTENCE
TelemetryBatch { host_id, timestamp, metrics: [{ metric_id, value }] }
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Push model (agents send batches to load balancers) allows auto-scaling endpoints; Pull model (Prometheus scraper) requires service discovery.