#09HardCore Distributed Systems

Design an API Rate Limiter

ASKED IN INTERVIEWS AT:AmazonAtlassianUberPatreon

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design a distributed, highly available rate limiter capable of enforcing tiered per-user and per-IP rate limits with atomic counters across distributed nodes.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS1 Million requests/sec across API gateway
READ / WRITE TRAFFIC RATIO1:1 read & increment on every request
5-YEAR STORAGE CAPACITYRedis RAM: ~20 bytes per user key * 100M active tokens = ~2 GB RAM

FUNCTIONAL REQUIREMENTS

  • [1]Throttle requests exceeding threshold (e.g. 100 req/min)
  • [2]Return HTTP 429 Too Many Requests with reset timestamp

NON-FUNCTIONAL REQUIREMENTS

  • •Sub-millisecond overhead to API requests
  • •Distributed synchronization across multiple edge proxy nodes
  • •Resilience to Redis downtime (fail open/closed)

CORE ARCHITECTURAL COMPONENTS

Envoy / NGINX API Gateway
Redis Cluster Cache
Lua Atomic Pipeline Scripts
Local In-Memory Cache (L1) with Sync

DATA SCHEMA & PERSISTENCE

Redis Hash: `rl:{userId}:{window}` -> { count, last_refill }

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Sliding Window Counter (accurate, low memory) vs Token Bucket (burst-tolerant, memory-efficient).