RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #34 // Very Hard
#34Very HardCore Distributed Systems

Design a Control Plane for a Distributed Database

ASKED IN INTERVIEWS AT:Netflix

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Architect the control plane for a cloud database fleet orchestrating node provisioning, automated shard rebalancing, leader elections, and zero-downtime upgrades.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPSOrchestrating 10,000+ database nodes across multiple cloud regions
READ / WRITE TRAFFIC RATIOLow frequency high-criticality operations
5-YEAR STORAGE CAPACITYetcd consensus cluster state store

FUNCTIONAL REQUIREMENTS

  • [1]Provision, scale, and decommission database clusters
  • [2]Automated failover on node health check failure
  • [3]Orchestrate live shard migrations

NON-FUNCTIONAL REQUIREMENTS

  • •Separation of control plane from data plane (data plane stays alive if control plane fails)
  • •No split-brain cluster states
  • •Auditable change log

CORE ARCHITECTURAL COMPONENTS

Control Plane API
etcd Cluster State Store
Topology Controller Reconciliation Loop
Node Lifecycle Agent
Health Monitor Sentinel

DATA SCHEMA & PERSISTENCE

ClusterSpec: cluster_id, desired_nodes, engine_version, shard_topology

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Declarative state with reconciliation loops ensures self-healing when real-world node states drift from desired configuration.