RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #10 // Very Hard
#10Very HardMonitoring, Telemetry & Logging
Design a Distributed Metrics Logging and Aggregation System
ASKED IN INTERVIEWS AT:GoogleMetaAmazoneBay
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Architect a multi-datacenter metrics pipeline (similar to Prometheus / Datadog) collecting billions of machine telemetry data points per second with downsampling and alerting.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS50,000 servers, 200 metrics/server every 10s = 1 Billion points/hour
READ / WRITE TRAFFIC RATIOHeavy write stream, bursty dashboard reads
5-YEAR STORAGE CAPACITYDouble-delta timestamp compression reduces raw storage by 90%
FUNCTIONAL REQUIREMENTS
- [1]Collect CPU, memory, and custom latency metrics from servers
- [2]Query aggregated metrics with p50, p95, p99 percentiles
- [3]Trigger alerts on SLA breaches
NON-FUNCTIONAL REQUIREMENTS
- •Handle 10M+ metrics/sec ingestion
- •Storage cost efficiency via automated downsampling
- •Real-time alerting (<10s latency)
CORE ARCHITECTURAL COMPONENTS
Host Agent (StatsD/OpenTelemetry)
Kafka Ingestion Queue
Flink Real-Time Aggregator
TSDB (ClickHouse / VictoriaMetrics)
Alert Evaluation Worker
DATA SCHEMA & PERSISTENCE
MetricKey: metric_name, labels_hash, timestamp_epoch, value_float64
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Raw metric retention (short-term 7 days) vs roll-ups into 1m/5m/1h buckets for long-term year trends.