RETURN TO SYSTEM DESIGN VAULT (55)BLUEPRINT #14 // Medium
#14MediumSearch, Autocomplete & Big Data
Design Web Crawler
ASKED IN INTERVIEWS AT:AmazonGoogleMetaAtlassian
EXECUTIVE SUMMARY & PROBLEM STATEMENT
Design a scalable distributed web crawler that traverses the World Wide Web, parses HTML content, filters duplicates with Bloom filters, and respects robots.txt politeness rules.
SCALE ESTIMATION & TRAFFIC ENVELOPE
DAILY ACTIVE USERS / TPS1 Billion web pages crawled per month = ~400 pages/sec
READ / WRITE TRAFFIC RATIOWrite-heavy document ingestion
5-YEAR STORAGE CAPACITY1 Billion pages * 100KB = 100 TB raw HTML/text monthly
FUNCTIONAL REQUIREMENTS
- [1]Given seed URLs, crawl and extract text and links
- [2]Store parsed pages in document store
- [3]Honor robots.txt and rate limits
NON-FUNCTIONAL REQUIREMENTS
- •High scalability (1 Billion pages/month)
- •Robustness against spider traps and infinite loops
- •Polite to target web hosts
CORE ARCHITECTURAL COMPONENTS
URL Frontier (Priority + Politeness Queues)
HTML Fetcher & Parser
Seen-URL Bloom Filter
Content Duplicate Detector (SimHash)
Document Storage (Bigtable/HBase)
DATA SCHEMA & PERSISTENCE
CrawlQueue (URL, priority, host_hash), HostPolitenessMap (host, last_crawled_at), DocumentStore (doc_id, url_hash, content, crawled_at)
DISTRIBUTED TRADE-OFFS & FAILURE MODES
- ⟡Breadth-first search gives broad coverage; Priority Queue (PageRank-biased) prioritizes high-value authority domains.