RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #11 // Very Hard
#11Very HardCore Distributed Systems
Design a Distributed Stream Processing System like Kafka
ASKED IN INTERVIEWS AT:AmazonMicrosoftWiseConfluent
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Design an append-only distributed commit log system providing partitioned, ordered event streams with high throughput, consumer group rebalancing, and zero-copy OS transfers.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPSEnterprise scale: 10M messages/sec throughput, 50TB daily journal writes
READ / WRITE TRAFFIC RATIOSequential write / sequential read
5-YEAR STORAGE CAPACITYOS PageCache utilization eliminates random disk I/O
FUNCTIONAL REQUIREMENTS
- [1]Publish events to topics
- [2]Distribute messages across topic partitions
- [3]Consumers subscribe and track stream offsets
NON-FUNCTIONAL REQUIREMENTS
- •Millions of messages per second throughput
- •Strict message ordering per partition
- •Fault-tolerant with configurable replicas
CORE ARCHITECTURAL COMPONENTS
Broker Nodes
Partition Leaders & In-Sync Replicas (ISR)
PageCache Buffers
Raft Consensus / Controller
Consumer Groups
DATA SCHEMA & PERSISTENCE
CommitLog Segments: offset (8 bytes), timestamp, payload_bytes
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Disk-backed sequential I/O with OS pagecache delivers 10x higher throughput than random-access in-memory message brokers.