RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #51 // Very Hard
#51Very HardCore Distributed Systems

Create a System to Migrate Large Data to Google Cloud

ASKED IN INTERVIEWS AT:GoogleOCI

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design an enterprise data migration pipeline moving 100 Petabytes of on-premises unstructured data to Google Cloud Storage with encryption, validation, and zero downtime.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS100 Petabytes across 10 Billion files
READ / WRITE TRAFFIC RATIOMassive one-way sequential migration stream
5-YEAR STORAGE CAPACITYCloud Storage standard bucket with tiered lifecycle policies

FUNCTIONAL REQUIREMENTS

  • [1]Transfer petabyte-scale files from on-premises to cloud storage
  • [2]Verify byte-for-byte data integrity
  • [3]Continuous synchronization of delta modifications

NON-FUNCTIONAL REQUIREMENTS

  • •Saturate available 100 Gbps dedicated network pipes
  • •Automated retry on packet drops without restarting entire files
  • •Zero impact on on-prem production workloads

CORE ARCHITECTURAL COMPONENTS

On-Premises Agent Cluster
Dedicated Cloud Interconnect (100 Gbps)
Parallel Transfer Gateway
GCS Object Storage
Verification & Reconciliation Worker

DATA SCHEMA & PERSISTENCE

MigrationManifest: file_path, size_bytes, crc32c_hash, status (PENDING, IN_TRANSIT, VERIFIED, FAILED)

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Combining physical transfer appliances (Google Transfer Appliance) for the initial 90PB baseline with network CDC sync for changes.