SQS)#

Q: How does a message queue work?

Producers publish messages to topics or queues backed by a durable log. Consumers read in order, tracking offsets so they can resume after restart and process messages reliably.

Q: What is the difference between Kafka, RabbitMQ, and SQS?

Kafka is a partitioned distributed log for high-throughput streaming. RabbitMQ is a traditional broker with rich routing. SQS is a fully managed queue with simple semantics.

Q: What is at-least-once vs exactly-once delivery?

At-least-once may deliver duplicates after retries and needs idempotent consumers. Exactly-once uses transactional writes and consumer offsets to make each message visible once.

Q: How do Kafka consumer groups work?

A consumer group divides partitions among its members so each partition is processed by exactly one consumer. Adding consumers rebalances partitions and scales throughput.

Q: Why does Kafka use a write-ahead log?

The WAL gives Kafka durable, ordered, append-only storage that supports replays, replication via follower fetches, and high sequential write throughput on commodity disks.

Q: How is Kafka replicated for high availability?

Each partition has a leader and follower replicas. Producers write to the leader, which waits for in-sync replicas to ack before acknowledging the produce request.

Problem statement (interviewer prompt)

Design Kafka (or RabbitMQ / SQS): producers publish messages to topics, consumers read in groups for parallelism. Provide partitioned ordering, configurable retention, at-least-once + exactly-once semantics, replication for HA, and tiered storage for cold data.

Apache Kafka logo, © Apache Software Foundation, via Wikimedia Commons.

flowchart LR
  P[Producer]
  B[[Broker]]
  T[(Topic / Partitions)]
  C[Consumer Group]
  P --> B --> T --> C

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class P,C service;
    class T datastore;
    class B queue;

flowchart TB
  subgraph Producers
    P1[App A]
    P2[App B]
    P3[CDC tail]
  end

  subgraph Brokers[Broker Cluster]
    direction TB
    META[Controller / KRaft<br/>metadata + leader election]
    B1[[Broker 1]]
    B2[[Broker 2]]
    B3[[Broker N]]
  end

  subgraph Topic
    P0[Partition 0]
    P1p[Partition 1]
    P2p[Partition 2]
    ISR[In-sync Replicas]
    LEAD[Leader / Followers]
  end

  subgraph Storage[Storage Layer]
    SEG[Segments + index]
    PAGE[Page cache]
    TIER[Tiered storage to S3]
    COMPACT[Log compaction by key]
    RETN[Retention by time/size]
  end

  subgraph Consumers
    G1[Group billing]
    G2[Group analytics]
    OFFSETS[(Committed offsets)]
    DLQ[[Dead-letter topic]]
  end

  subgraph Semantics
    AL[at-least-once]
    EOS[Exactly-once via tx]
    ORD[Per-partition order]
    KEY[Partition by key]
  end

  Producers --> Brokers
  Brokers --> Topic
  Topic --> Storage
  Topic --> Consumers
  Consumers --> OFFSETS
  Consumers --> DLQ
  Semantics --- Topic

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class P1,P2,P3,META,P0,P1p,P2p,ISR,LEAD,SEG,COMPACT,RETN,G1,G2,AL,EOS,ORD,KEY service;
    class OFFSETS datastore;
    class PAGE cache;
    class B1,B2,B3,DLQ queue;
    class TIER storage;

Replication & ISR#

Each partition has one leader, N followers.
Producers write to leader; followers fetch.
ISR = followers caught up within replica.lag.time.max.ms.
Producer acks=all waits for ISR commit; acks=1 waits for leader only.

Consumer groups#

Each partition is consumed by exactly one consumer in a group.
Rebalance on member add/remove (causes brief pause).
Offsets stored in __consumer_offsets; consumers commit periodically.

Exactly-once semantics#

Idempotent producer (sequence number per partition).
Transactional producer for write-and-commit-offset atomically.
Consumer reads read_committed isolation.

Tiered storage#

Hot segments local; older segments moved to S3.
Brokers read transparently from S3 when needed.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
`HLD`	Raft / Paxos consensus	replicated state machine via majority quorum	consensus-raft-paxos
`HLD`	Leader/follower replication	sync/semi-sync/async replication, failover	replication-leader-follower
`HLD`	LSM vs B-Tree engines	WAL, memtable, SSTables, compaction	storage-engines-lsm-btree
`HLD`	Change Data Capture	WAL/binlog tailing, outbox publishing	change-data-capture
`LLD`	Testing strategy	pyramid, doubles, TDD, contracts	testing-strategy
`LLD`	Concurrency primitives	mutex, semaphore, RW lock, atomic, CAS	concurrency-primitives

Quick reference#

Functional#

Persistent log of records by topic.
Partitioned for parallelism.
Multiple consumer groups read independently.
Ordering per partition; retention by time/size.

Non-functional#

1M+ msg/s/broker possible (Kafka, with batching).
p99 produce < 10 ms with batched producer.
99.99% durability with RF=3 + acks=all.

Capacity#

Disk = throughput × retention; e.g., 100 MB/s × 7 days × 3 replicas = 180 TB cluster.
Network often the bottleneck.

API#

Produce(topic, key?, value, headers)
Consume(topic, group, partition, offset)
CommitOffset(group, partition, offset)

Trade-offs#

Push (RabbitMQ) vs Pull (Kafka): pull self-regulates, push tightens latency.
EOS has cost (transactional overhead); use only where needed.
Tiered storage trades S3 latency for far longer retention.

Refs#

Kafka design doc (Confluent), "I Heart Logs" Jay Kreps, RabbitMQ docs, NATS JetStream, Pulsar architecture.

FAQ#

How does a message queue work?#