Skip to content

Stream processors - Flink vs Kafka Streams vs Spark Structured Streaming#

The three engines you'll be asked to pick between when designing a streaming pipeline. They differ on latency floor, state model, and operational shape.

Side-by-side#

Apache Flink Kafka Streams Spark Structured Streaming
Execution model true streaming true streaming (library) micro-batch (sub-second possible)
Latency floor < 100 ms < 100 ms seconds (recently sub-second)
State backend RocksDB or in-mem, checkpointed RocksDB (local) state store, checkpointed
Deployment own cluster (k8s / YARN) embedded library in your JVM app own cluster (Spark)
Joins (stream × stream, stream × table) first-class first-class improving
Watermarks / event-time first-class first-class first-class
Exactly-once yes (transactional sinks) yes (Kafka transactions) yes (idempotent / transactional sinks)
Backpressure built-in (credit-based) natural (Kafka consumer pull) built-in
Language Java / Scala / Python / SQL Java / Scala (JVM) Scala / Java / Python / SQL
Sweet spot latency-critical analytics, complex CEP, heavy joins "Kafka in, Kafka out" stream apps; small ops footprint batch + stream in one code base; big data lake integration

Quick chooser#

flowchart TB
  Q[Need stream processing]
  Q --> A{Already on Kafka,<br/>small footprint?}
  A -->|yes| KS[Kafka Streams - library in your app]
  A -->|no| B{Sub-second latency<br/>+ complex stateful joins?}
  B -->|yes| FL[Flink]
  B -->|no| SP[Spark Structured Streaming]

  classDef p fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef s fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  class Q,A,B p;
  class KS,FL,SP s;

When each wins#

  • Flink - lowest latency floor, richest state APIs, complex event processing (CEP), heavy stream-stream joins, savepoint-driven upgrades. Run on its own cluster.
  • Kafka Streams - when you already speak Kafka, want a library (not a cluster), and your topology fits "topic → transforms → topic". No separate ops surface - it's just a library in your JVM service.
  • Spark Structured Streaming - when you want batch + stream in one codebase, integration with the data lake (Iceberg / Delta / Hudi), and seconds-not-milliseconds is fine.

Shared building blocks (you must know these regardless)#

  • Event time vs processing time - when the event happened vs when you processed it.
  • Watermark - "we've seen all events with event_time ≤ T". Triggers window close.
  • Windowing - tumbling (fixed, non-overlapping), hopping (sliding, overlap), session (gap-defined).
  • State - keyed state, list state, value state; backed by RocksDB + checkpoints.
  • Checkpoints - periodic durable snapshot; on failure, restart from latest checkpoint.
  • Exactly-once - needs replayable source + idempotent / transactional sink + deterministic operator.

Common interview hooks#

  • "What's the difference between event time and processing time?" → see above; processing-time windows lose data on backfill.
  • "How do watermarks handle late events?" → grace period after the watermark advances; configurable.
  • "Why Flink over Spark for low-latency?" → Spark micro-batches (even at 50-500 ms); Flink processes per-event.
  • "Reprocessing 6 months of data?" → with Kappa, replay from Kafka retention or tiered storage; with Lambda, run the batch path.

Glossary & fundamentals#

Tag Concept Page
HLD Batch & stream processing (Lambda vs Kappa) batch-stream-processing
HLD Pub/Sub (Kafka usually the bus) pub-sub-pattern
HLD Logical clocks (event time) logical-clocks
HLD Idempotency + retries (sinks) idempotency-retries

FAQ#

Flink is a standalone cluster engine for complex stateful streaming with sub-100ms latency. Kafka Streams is an embedded JVM library that reads and writes Kafka topics with minimal ops footprint.

Is Spark Structured Streaming real time?#

Spark Structured Streaming uses a micro-batch model and historically delivered seconds-level latency. Recent versions support continuous processing with sub-second latency, but Flink stays lower at the tail.

Pick Flink when you need true event-time streaming, complex event processing, large keyed state, or sub-100ms latency. Pick Spark when you want one engine for batch and stream with a unified API.

Does Kafka Streams support exactly once?#

Yes, Kafka Streams achieves exactly-once semantics by using Kafka transactions to atomically commit input offsets, state changes, and output records together within a single transaction.

Which stream processor is best for system design interviews?#

Mention all three by their sweet spot: Flink for latency-critical analytics and CEP, Kafka Streams for Kafka-in Kafka-out apps, and Spark when batch and stream share code. Naming tradeoffs scores higher than picking one.

Further reading#