#29HardSearch, Autocomplete & Big Data

Design an A/B Testing System (like Optimizely)

ASKED IN INTERVIEWS AT:AffirmFAANG

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design an enterprise experimentation platform supporting multi-layered A/B tests, deterministic bucket hashing, statistical significance calculations, and zero client latency degradation.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS100 Million daily active users across thousands of active experiments
READ / WRITE TRAFFIC RATIOContinuous analytics write stream
5-YEAR STORAGE CAPACITYExperiment configurations distributed via CDN to clients

FUNCTIONAL REQUIREMENTS

  • [1]Assign users consistently to test variants (A, B, or C)
  • [2]Support targeting rules (country, device, user attributes)
  • [3]Track metrics and compute conversion p-values

NON-FUNCTIONAL REQUIREMENTS

  • •Zero network latency for user bucket evaluation (evaluate in client SDK)
  • •Deterministic bucketing without storing user assignments in DB
  • •Zero test interference

CORE ARCHITECTURAL COMPONENTS

Experiment Config Management UI
Global CDN Config Distributor
In-Memory Client SDK Evaluator
Clickstream Exposure Event Logger
Analysis Engine (Spark)

DATA SCHEMA & PERSISTENCE

Experiments: experiment_id, layer_id, traffic_allocation_percent, salt, variants: [{ name, weight }]

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Hashing `hash(userId + experimentSalt) % 100` guarantees deterministic assignment without requiring database lookups.