#17HardMonitoring, Telemetry & Logging

System to Collect Performance Metrics from Thousands of Servers

ASKED IN INTERVIEWS AT:GoogleDatadogAmazoneBay

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Build a telemetry collection architecture aggregating system utilization metrics (CPU, disk, network, JVM) from 100,000+ nodes without overwhelming target networks.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS100,000 host nodes * 50 metrics * every 5s = 1 Million metrics/sec
READ / WRITE TRAFFIC RATIOPure continuous write stream
5-YEAR STORAGE CAPACITYCompressed protocol buffer payloads: ~50 MB/sec ingestion

FUNCTIONAL REQUIREMENTS

  • [1]Lightweight agent collects OS metrics every 5 seconds
  • [2]Buffer and stream metrics to central aggregation cluster
  • [3]Detect dead or failing hosts

NON-FUNCTIONAL REQUIREMENTS

  • •Agent CPU overhead < 1%
  • •Resilient to network partitions (local agent spooling)
  • •Support 100K+ concurrent agents

CORE ARCHITECTURAL COMPONENTS

Local Host Daemon (eBPF / Go agent)
Load Balancer Layer
Kafka Stream Buffer
Aggregator Workers
Time-Series Long Term Store

DATA SCHEMA & PERSISTENCE

TelemetryBatch { host_id, timestamp, metrics: [{ metric_id, value }] }

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Push model (agents send batches to load balancers) allows auto-scaling endpoints; Pull model (Prometheus scraper) requires service discovery.