RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #41 // Very Hard
#41Very HardMonitoring, Telemetry & Logging
Design a Distributed Tracing System
ASKED IN INTERVIEWS AT:UberAmazon
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Design an end-to-end distributed tracing platform (like Jaeger / Dapper / Zipkin) tracking request spans and causal DAG relationships across thousands of microservices.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS50 Billion spans generated daily across 1,000 microservices
READ / WRITE TRAFFIC RATIOHeavy write stream, selective troubleshooting reads
5-YEAR STORAGE CAPACITY1% head sampling + 100% error tail sampling reduces storage by 95%
FUNCTIONAL REQUIREMENTS
- [1]Trace requests flowing across dozens of asynchronous microservices
- [2]Collect timing spans, logs, and error tags per service
- [3]Render interactive latency flame graphs
NON-FUNCTIONAL REQUIREMENTS
- •Negligible overhead on application runtime (< 0.5%)
- •Adaptive sampling to capture 100% of errors while discarding mundane traces
- •Petabyte scalability
CORE ARCHITECTURAL COMPONENTS
OpenTelemetry Tracer SDK
Trace Context Injector/Extractor
Kafka Collector Queue
Tail Sampling Collector
Columnar Trace Store (ClickHouse)
DATA SCHEMA & PERSISTENCE
Span: trace_id (16B), span_id (8B), parent_span_id, service_name, start_time, duration_us, tags_json
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Tail-based sampling buffers spans in memory before deciding to keep them, ensuring 100% of error traces and slow requests are saved.