RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #10 // Very Hard
#10Very HardMonitoring, Telemetry & Logging

Design a Distributed Metrics Logging and Aggregation System

ASKED IN INTERVIEWS AT:GoogleMetaAmazoneBay

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Architect a multi-datacenter metrics pipeline (similar to Prometheus / Datadog) collecting billions of machine telemetry data points per second with downsampling and alerting.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS50,000 servers, 200 metrics/server every 10s = 1 Billion points/hour
READ / WRITE TRAFFIC RATIOHeavy write stream, bursty dashboard reads
5-YEAR STORAGE CAPACITYDouble-delta timestamp compression reduces raw storage by 90%

FUNCTIONAL REQUIREMENTS

  • [1]Collect CPU, memory, and custom latency metrics from servers
  • [2]Query aggregated metrics with p50, p95, p99 percentiles
  • [3]Trigger alerts on SLA breaches

NON-FUNCTIONAL REQUIREMENTS

  • •Handle 10M+ metrics/sec ingestion
  • •Storage cost efficiency via automated downsampling
  • •Real-time alerting (<10s latency)

CORE ARCHITECTURAL COMPONENTS

Host Agent (StatsD/OpenTelemetry)
Kafka Ingestion Queue
Flink Real-Time Aggregator
TSDB (ClickHouse / VictoriaMetrics)
Alert Evaluation Worker

DATA SCHEMA & PERSISTENCE

MetricKey: metric_name, labels_hash, timestamp_epoch, value_float64

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Raw metric retention (short-term 7 days) vs roll-ups into 1m/5m/1h buckets for long-term year trends.