Message brokers - Kafka vs RabbitMQ vs SQS vs Pulsar#

The most-asked "which queue should I use?" question in interviews. Each row is a meaningful axis of difference; pick the broker that matches your dominant requirement.

Side-by-side#

	Apache Kafka	RabbitMQ	AWS SQS	Apache Pulsar
Mental model	append-only log, partitioned	exchange + queues, AMQP	managed queue	log + queues, segmented storage
Ordering	per partition	per queue (with care)	best-effort (FIFO opt)	per partition / key
Delivery	at-least-once + EOS option	at-least-once	at-least-once	at-least-once + EOS option
Throughput	very high (MB/s per partition)	medium	high (managed)	very high
Retention	days-months on disk	until consumed	up to 14 days	unbounded (tiered to S3)
Replay	yes (offset-based)	no	no	yes
Push vs pull	pull (consumer poll)	push (broker pushes)	pull (long-poll)	both
Fan-out	consumer groups	exchanges + bindings	SNS → SQS fan-out	subscriptions + multi-topic
Backpressure	natural (pull-based)	manual / prefetch	natural	natural
Geo-replication	MirrorMaker / Confluent	federation / shovel	cross-region replication	built-in geo replication
Ops complexity	high (ZK / KRaft cluster)	medium	none (managed)	high (BookKeeper + brokers)
Multi-tenant isolation	per-topic ACLs	vhosts	per-account / queue	namespaces + tenants
Schema	external (Schema Registry)	none	none	built-in (Avro, JSON, Protobuf)
Best for	high-throughput log / streaming	classic task queue / RPC	hands-off async work	log + queue hybrid, geo-active

Quick chooser#

flowchart TB
  Q[Need a message broker]
  Q --> A{Cloud-only,<br/>hands-off?}
  A -->|yes| SQS[Use SQS / SNS]
  A -->|no| B{Streaming,<br/>replay, very high throughput?}
  B -->|yes| C{Built-in geo<br/>replication?}
  C -->|yes| PULSAR[Use Pulsar]
  C -->|no| KAFKA[Use Kafka]
  B -->|no| RABBIT[Use RabbitMQ]

  classDef p fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef s fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  class Q,A,B,C p;
  class SQS,KAFKA,RABBIT,PULSAR s;

When each wins#

Kafka - event-streaming backbone, CDC pipelines, replay for analytics, exactly-once across topic boundaries, building a "log-of-everything" foundation.
RabbitMQ - classic task queue, low-latency RPC patterns, complex routing via exchanges (topic, fan-out, headers), small teams that want minimal ops.
SQS - anything in AWS where you just want "drop a message, pick it up later" without operating Kafka. Pair with SNS for fan-out.
Pulsar - when you want both Kafka-style streaming and RabbitMQ-style queue semantics, plus first-class geo-replication and tiered storage. Best where ops can absorb BookKeeper.

Performance envelopes (rough)#

Broker	Single-broker throughput	p99 latency target
Kafka	10-100k msgs/s per partition × N partitions	5-20 ms
RabbitMQ	30-50k msgs/s (durable)	1-5 ms
SQS	3000 msgs/s standard; 300k+ with batching	tens of ms (managed)
Pulsar	100k+ msgs/s per topic	5-15 ms

Operational gotchas#

Kafka: rebalance storms during consumer add/remove; ZK → KRaft migration; ISR drift; hot partitions.
RabbitMQ: queue memory bloat under slow consumers; mirrored queues are deprecated → use Quorum Queues; classic clustering quirks.
SQS: visibility-timeout misconfiguration causing duplicate processing; 256 KB message size cap; long-poll vs short-poll.
Pulsar: BookKeeper bookies as a separate state plane to operate; ZooKeeper still in the loop; learning curve.

Glossary & fundamentals#

Tag	Concept	Page
`HLD`	Pub/Sub & message brokers	pub-sub-pattern
`HLD`	Idempotency + retries (consumer-side)	idempotency-retries
`HLD`	Change Data Capture (kafka often the bus)	change-data-capture
`HLD`	Event sourcing + CQRS	event-sourcing-cqrs
`HLD`	Batch & stream processing	batch-stream-processing

FAQ#

Should I use Kafka or RabbitMQ?#

Use Kafka for event streaming, replay, and high-volume ordered logs. Use RabbitMQ when you need flexible routing with exchanges, per-message acks, and easy work-queue semantics for task processing.

What is the difference between Kafka and SQS?#

Kafka is a partitioned append-only log with durable replay. SQS is a managed queue with at-least-once delivery and no replay. SQS is cheaper to operate but cannot replace Kafka for stream processing.

When should I pick Pulsar over Kafka?#

Pulsar wins when you need multi-tenant isolation, geo-replication out of the box, and separate storage and compute. Kafka has a larger ecosystem and tooling, so it is the safer default for most teams.

Does RabbitMQ guarantee ordering?#

RabbitMQ preserves order within a single queue when there is one consumer. Multiple consumers, requeues, and priority queues can reorder messages, so do not rely on global ordering across the broker.

Can I use SQS for event sourcing?#

Not directly. SQS lacks replay and ordered partitions. Pair SQS with EventBridge or use Kinesis for log-style event sourcing on AWS, or stand up MSK if you need full Kafka semantics.

Video walkthrough

Kafka System Design Deep Dive : via Hello Interview