RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #41 // Very Hard
#41Very HardMonitoring, Telemetry & Logging

Design a Distributed Tracing System

ASKED IN INTERVIEWS AT:UberAmazon

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design an end-to-end distributed tracing platform (like Jaeger / Dapper / Zipkin) tracking request spans and causal DAG relationships across thousands of microservices.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS50 Billion spans generated daily across 1,000 microservices
READ / WRITE TRAFFIC RATIOHeavy write stream, selective troubleshooting reads
5-YEAR STORAGE CAPACITY1% head sampling + 100% error tail sampling reduces storage by 95%

FUNCTIONAL REQUIREMENTS

  • [1]Trace requests flowing across dozens of asynchronous microservices
  • [2]Collect timing spans, logs, and error tags per service
  • [3]Render interactive latency flame graphs

NON-FUNCTIONAL REQUIREMENTS

  • •Negligible overhead on application runtime (< 0.5%)
  • •Adaptive sampling to capture 100% of errors while discarding mundane traces
  • •Petabyte scalability

CORE ARCHITECTURAL COMPONENTS

OpenTelemetry Tracer SDK
Trace Context Injector/Extractor
Kafka Collector Queue
Tail Sampling Collector
Columnar Trace Store (ClickHouse)

DATA SCHEMA & PERSISTENCE

Span: trace_id (16B), span_id (8B), parent_span_id, service_name, start_time, duration_us, tags_json

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Tail-based sampling buffers spans in memory before deciding to keep them, ensuring 100% of error traces and slow requests are saved.