#13HardSearch, Autocomplete & Big Data

Identify the K Most Shared Articles in Various Time Windows

ASKED IN INTERVIEWS AT:LinkedInMetaTwitter

EXECUTIVE SUMMARY & PROBLEM STATEMENT

Design a real-time analytics streaming engine that computes the top K most viral/shared articles across sliding time windows (past 5 minutes, 1 hour, 24 hours).

SCALE ESTIMATION & TRAFFIC ENVELOPE

DAILY ACTIVE USERS / TPS500 Million share events per day = ~6,000 events/sec
READ / WRITE TRAFFIC RATIOHigh volume event ingestion with cached dashboard reads
5-YEAR STORAGE CAPACITYApproximate probabilistic counters fit in <500 MB RAM

FUNCTIONAL REQUIREMENTS

  • [1]Track share events per article ID
  • [2]Query top K articles across 5min, 1hr, 24hr sliding windows

NON-FUNCTIONAL REQUIREMENTS

  • •Real-time window update latency < 5s
  • •Memory bounded regardless of millions of unique articles
  • •High availability

CORE ARCHITECTURAL COMPONENTS

Event Ingestion Gateway
Kafka Event Stream
Flink Sliding Window State Machine
Redis Top-K Cache
Dashboard API

DATA SCHEMA & PERSISTENCE

Count-Min Sketch array + Min-Heap storing `{ article_id, frequency }`

DISTRIBUTED TRADE-OFFS & FAILURE MODES

  • ⟡Probabilistic data structures (Count-Min Sketch) trade 1% counting precision for 99% RAM reduction.