RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #34 // Very Hard
#34Very HardCore Distributed Systems
Design a Control Plane for a Distributed Database
ASKED IN INTERVIEWS AT:Netflix
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Architect the control plane for a cloud database fleet orchestrating node provisioning, automated shard rebalancing, leader elections, and zero-downtime upgrades.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPSOrchestrating 10,000+ database nodes across multiple cloud regions
READ / WRITE TRAFFIC RATIOLow frequency high-criticality operations
5-YEAR STORAGE CAPACITYetcd consensus cluster state store
FUNCTIONAL REQUIREMENTS
- [1]Provision, scale, and decommission database clusters
- [2]Automated failover on node health check failure
- [3]Orchestrate live shard migrations
NON-FUNCTIONAL REQUIREMENTS
- •Separation of control plane from data plane (data plane stays alive if control plane fails)
- •No split-brain cluster states
- •Auditable change log
CORE ARCHITECTURAL COMPONENTS
Control Plane API
etcd Cluster State Store
Topology Controller Reconciliation Loop
Node Lifecycle Agent
Health Monitor Sentinel
DATA SCHEMA & PERSISTENCE
ClusterSpec: cluster_id, desired_nodes, engine_version, shard_topology
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Declarative state with reconciliation loops ensures self-healing when real-world node states drift from desired configuration.