RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #19 // Hard
#19HardCore Distributed Systems
Design a Distributed Queue like RabbitMQ
ASKED IN INTERVIEWS AT:AmazonAppleInstacart
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Architect a distributed message broker providing AMQP-style exchange routing, dead-letter queues, message deduplication, and consumer acknowledgments.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS500K messages/second throughput
READ / WRITE TRAFFIC RATIO1:1 publish-to-consume
5-YEAR STORAGE CAPACITYTransient in-memory queues with disk spooling on memory watermark (80%)
FUNCTIONAL REQUIREMENTS
- [1]Publish message to exchange with routing key
- [2]Consumers consume from bound queues with explicit ACK
- [3]Dead-letter message after N retries
NON-FUNCTIONAL REQUIREMENTS
- •Guaranteed delivery under broker node crash
- •Zero message loss with persistent storage
- •Fair consumer dispatching
CORE ARCHITECTURAL COMPONENTS
Protocol Gateway (AMQP / TCP)
Exchange Router
Queue Manager (Ring Buffer)
Disk Wal Persistence
Acknowledgment Coordinator
DATA SCHEMA & PERSISTENCE
MessageRecord: msg_id, payload, routing_key, priority, attempts, state
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡In-memory queues with asynchronous flush give high throughput but risk losing un-flushed messages; synchronous fsync ensures durability at 10x lower throughput.