Twitter / X#

Q: How does Twitter generate the home timeline?

Fan-out on write pushes each tweet into Redis ZSETs of followers at publish time. Reading the timeline is then a single ZRANGE call, hitting p99 under 100 ms.

Q: How does Twitter handle celebrity accounts?

Celebrities skip fan-out and their tweets are merged at read time. The reader joins their pushed timeline with fresh tweets from a small set of followed celebrities.

Q: Why use fan-out on write instead of on read?

Read traffic dominates by 100x. Doing the work at write time gives the hot path a single cache call instead of expensive cross-shard scatter-gather on every feed open.

Q: How is tweet storage sharded?

Tweets are stored in an LSM-tree KV like Manhattan or RocksDB, sharded by tweet ID. The social graph and timeline caches live in separate fleets tuned to their access patterns.

Q: How does Twitter search find tweets so fast?

Earlybird inverted indexes are sharded by time. Recent tweets sit in memory, older ones tier to disk. A query fans out across shards, merges results, and ranks them.

Problem statement (interviewer prompt)

Design Twitter/X: users post 280-char tweets, follow each other, and see a home feed of tweets from people they follow (chronological + For-You). Support 500M tweets/day, 5B reads/day, hashtag search, trends, likes, retweets, replies, and DMs.

flowchart LR
  U([User])
  TW[Tweet Service]
  FAN[[Fan-out]]
  TL[(Home Timelines<br/>Redis ZSET)]
  TWDB[(Tweet DB)]
  GRAPH[(Social Graph)]
  HOME[Home Timeline API]
  SEARCH[Search]
  U -->|post| TW --> TWDB
  TW --> FAN
  GRAPH --> FAN
  FAN --> TL
  U -->|read| HOME --> TL
  TWDB -. index .-> SEARCH

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class U client;
    class TW,HOME,SEARCH service;
    class TWDB datastore;
    class TL cache;
    class FAN queue;

flowchart TB
  subgraph Clients
    iOS[iOS]
    AND[Android]
    WEB([Web])
  end

  subgraph Edge
    DNS[DNS]
    CDN[CDN media + assets]
    LB[L7 LB]
    GW[API Gateway / GraphQL]
    WAF[WAF + Bot]
  end

  subgraph Write[Write Path]
    TWS[Tweet Service]
    MS[Media Service]
    OBJ[(S3 / blob)]
    ID[Snowflake ID]
    INGEST[[Kafka tweet stream]]
  end

  subgraph Graph[Social Graph]
    FG[Follow Service<br/>FlockDB / TAO-like]
    GDB[(Graph store)]
  end

  subgraph FanOut[Fanout - Hybrid]
    FANW[[Fanout workers]]
    CELEB([Celebrity classifier])
    PUSH[[Push fanout<br/>writes per follower]]
    PULL[Pull on read<br/>for celeb followees]
    HOMECACHE[(Home Timeline cache<br/>Redis ZSET per user)]
  end

  subgraph Storage
    TDB[(Tweets store<br/>Manhattan / Cassandra)]
    USERDB[(User store)]
    LIKES[(Likes / Retweets KV)]
    TRENDS[(Trends KV)]
  end

  subgraph Read[Read Path]
    HOME[Home Timeline API]
    USER([User Timeline API])
    HYD([Hydrator<br/>tweet + author + counts])
    RANK([ML Ranker<br/>For-You])
    DEDUP[Dedup + filter]
  end

  subgraph Search[Search / Trends]
    EARLY[Earlybird<br/>real-time inverted index]
    SPARK([Spark / Heron streams])
    QPARSE([Query parser])
  end

  subgraph Media
    TRANS([Transcoder])
    THUMB[Thumbnailer]
    LIVE[Live streaming<br/>Periscope-style]
  end

  subgraph Notif
    NS[Notification Service]
    PUSHN((APNS / FCM))
  end

  subgraph ML
    REC([Recommendation<br/>Who-to-follow])
    SCORE[Scoring service]
    SAFE[Trust & Safety<br/>spam / abuse models]
  end

  subgraph Obs
    MET[Metrics]
    TRC[Trace]
    LOG[Logs]
  end

  Clients --> DNS --> CDN --> LB --> WAF --> GW
  GW --> TWS
  TWS --> ID
  TWS --> TDB
  TWS --> INGEST
  TWS --> MS --> OBJ
  MS --> TRANS --> THUMB
  INGEST --> FANW
  FG --> FANW
  CELEB --> FANW
  FANW -->|normal user| PUSH --> HOMECACHE
  FANW -->|celeb author| PULL
  GW --> HOME --> HOMECACHE
  HOME --> PULL --> TDB
  HOME --> HYD --> RANK --> DEDUP --> Clients
  USER --> TDB
  INGEST --> EARLY
  GW --> Search --> EARLY
  INGEST --> SPARK --> TRENDS
  TWS --> LIKES
  INGEST --> NS --> PUSHN
  SAFE -.filter.-> RANK
  REC -.suggestions.-> Clients

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class WEB,USER client;
    class DNS,CDN,LB,GW,WAF edge;
    class iOS,AND,TWS,MS,PULL,HOME,DEDUP,THUMB,LIVE,NS,SCORE,SAFE service;
    class ID,FG,GDB,TDB,USERDB,LIKES,TRENDS,EARLY datastore;
    class HOMECACHE cache;
    class INGEST,FANW,PUSH queue;
    class CELEB,HYD,RANK,SPARK,QPARSE,TRANS,REC compute;
    class OBJ storage;
    class PUSHN external;
    class MET,TRC,LOG obs;

Hybrid fan-out#

Normal user (<10k followers): fan-out push to timeline cache.
Celebrity (>1M followers): no fan-out; pull at read for that subset.
Merger interleaves push-cached tweets with celeb tweets by timestamp.

Storage#

Tweets: Manhattan (Twitter's KV) - tweet_id PK, denormalized counts.
Likes/RTs: counters (TAO-style or Redis HLL for view counts).
Search: Earlybird = Lucene-based real-time inverted index, segmented by time.

Ranking#

For-You feed = candidates from network + recommendations → heavy DNN scorer.
Latency budget per scorer < 10 ms.

Live tweets#

Read-after-write: route subsequent reads to coordinator that committed.
Push to active sessions over Streaming API (Server-Sent Events / WS).

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Load balancer / GSLB	L4/L7 traffic distribution and failover	load-balancer
`HLD`	CDN	edge caching for static assets	cdn
`HLD`	API gateway / BFF	single ingress, auth, rate limit, routing	api-gateway
`HLD`	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
`HLD`	Probabilistic data structures	Bloom, HLL, Count-Min, MinHash, t-digest	probabilistic-data-structures
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`HLD`	Realtime protocols	WS / SSE / polling / gRPC streaming	realtime-protocols
`HLD`	Search internals	inverted index, BM25, embeddings, ANN	search-internals

Quick reference#

Functional#

Post tweet (text/media, ≤ 280 chars).
Follow / unfollow.
Home timeline (chronological + For-You).
Search, hashtag/trend, mentions.
Like, retweet, reply, DM.
Notifications.

Non-functional#

500M tweets/day, 5B reads/day.
p99 home timeline open < 200 ms.
99.99% availability.

Capacity#

500M tweets/day → 5,800/s avg, 50k/s peak.
5B reads/day → 58k/s avg, 500k/s peak.
Average follower count ~200; fan-out write = 500M × 200 = 100B/day = 1M/s.
Tweet storage: 500M × 365 × 1 KB = 180 TB/yr text + media in S3.

Schema#

tweets(id PK, author_id, text, created_at, lang, media_ids[])
follow(follower_id, followee_id, created_at) (sharded by follower_id and by followee_id - both directions).
home_timeline(user_id, [(ts, tweet_id), ...]) - Redis ZSET, capped 800.
likes(tweet_id, user_id); counts in Redis.

Trade-offs#

Push wins for read latency but explodes for celebs.
Pull-only simpler but adds latency and load to author shards.
Hybrid is the production answer; merger adds complexity.
Ranked feed (ML) drives engagement but at compute cost and feedback-loop hazards.

Refs#

"The infrastructure behind Twitter" engineering blog, FlockDB paper, Manhattan KV blog, Earlybird real-time search paper, ByteByteGo "Design Twitter", Alex Xu Vol 2.

FAQ#

How does Twitter generate the home timeline?#