RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #11 // Very Hard
#11Very HardCore Distributed Systems

Design a Distributed Stream Processing System like Kafka

ASKED IN INTERVIEWS AT:AmazonMicrosoftWiseConfluent

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design an append-only distributed commit log system providing partitioned, ordered event streams with high throughput, consumer group rebalancing, and zero-copy OS transfers.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPSEnterprise scale: 10M messages/sec throughput, 50TB daily journal writes
READ / WRITE TRAFFIC RATIOSequential write / sequential read
5-YEAR STORAGE CAPACITYOS PageCache utilization eliminates random disk I/O

FUNCTIONAL REQUIREMENTS

  • [1]Publish events to topics
  • [2]Distribute messages across topic partitions
  • [3]Consumers subscribe and track stream offsets

NON-FUNCTIONAL REQUIREMENTS

  • •Millions of messages per second throughput
  • •Strict message ordering per partition
  • •Fault-tolerant with configurable replicas

CORE ARCHITECTURAL COMPONENTS

Broker Nodes
Partition Leaders & In-Sync Replicas (ISR)
PageCache Buffers
Raft Consensus / Controller
Consumer Groups

DATA SCHEMA & PERSISTENCE

CommitLog Segments: offset (8 bytes), timestamp, payload_bytes

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Disk-backed sequential I/O with OS pagecache delivers 10x higher throughput than random-access in-memory message brokers.