RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #14 // Medium
#14MediumSearch, Autocomplete & Big Data

Design Web Crawler

ASKED IN INTERVIEWS AT:AmazonGoogleMetaAtlassian

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design a scalable distributed web crawler that traverses the World Wide Web, parses HTML content, filters duplicates with Bloom filters, and respects robots.txt politeness rules.

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS1 Billion web pages crawled per month = ~400 pages/sec
READ / WRITE TRAFFIC RATIOWrite-heavy document ingestion
5-YEAR STORAGE CAPACITY1 Billion pages * 100KB = 100 TB raw HTML/text monthly

FUNCTIONAL REQUIREMENTS

  • [1]Given seed URLs, crawl and extract text and links
  • [2]Store parsed pages in document store
  • [3]Honor robots.txt and rate limits

NON-FUNCTIONAL REQUIREMENTS

  • •High scalability (1 Billion pages/month)
  • •Robustness against spider traps and infinite loops
  • •Polite to target web hosts

CORE ARCHITECTURAL COMPONENTS

URL Frontier (Priority + Politeness Queues)
HTML Fetcher & Parser
Seen-URL Bloom Filter
Content Duplicate Detector (SimHash)
Document Storage (Bigtable/HBase)

DATA SCHEMA & PERSISTENCE

CrawlQueue (URL, priority, host_hash), HostPolitenessMap (host, last_crawled_at), DocumentStore (doc_id, url_hash, content, crawled_at)

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Breadth-first search gives broad coverage; Priority Queue (PageRank-biased) prioritizes high-value authority domains.