RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #51 // Very Hard
#51Very HardCore Distributed Systems
Create a System to Migrate Large Data to Google Cloud
ASKED IN INTERVIEWS AT:GoogleOCI
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Design an enterprise data migration pipeline moving 100 Petabytes of on-premises unstructured data to Google Cloud Storage with encryption, validation, and zero downtime.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS100 Petabytes across 10 Billion files
READ / WRITE TRAFFIC RATIOMassive one-way sequential migration stream
5-YEAR STORAGE CAPACITYCloud Storage standard bucket with tiered lifecycle policies
FUNCTIONAL REQUIREMENTS
- [1]Transfer petabyte-scale files from on-premises to cloud storage
- [2]Verify byte-for-byte data integrity
- [3]Continuous synchronization of delta modifications
NON-FUNCTIONAL REQUIREMENTS
- •Saturate available 100 Gbps dedicated network pipes
- •Automated retry on packet drops without restarting entire files
- •Zero impact on on-prem production workloads
CORE ARCHITECTURAL COMPONENTS
On-Premises Agent Cluster
Dedicated Cloud Interconnect (100 Gbps)
Parallel Transfer Gateway
GCS Object Storage
Verification & Reconciliation Worker
DATA SCHEMA & PERSISTENCE
MigrationManifest: file_path, size_bytes, crc32c_hash, status (PENDING, IN_TRANSIT, VERIFIED, FAILED)
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Combining physical transfer appliances (Google Transfer Appliance) for the initial 90PB baseline with network CDC sync for changes.