Skip to content

Lambda and Kappa Architecture#

Problem statement (interviewer prompt)

A user-behaviour analytics platform must serve dashboards with both recent (sub-minute) and historical (multi-year, fully reprocessed) views. Compare two reference architectures: dual batch+stream (Lambda) and stream-only (Kappa). When should you pick each?

Two competing reference architectures for analytics:

flowchart LR
  subgraph Lambda
    direction LR
    Ev1[Events] --> Batch[Batch layer<br/>Spark / Hive]
    Ev1 --> Speed[Speed layer<br/>Flink / Storm]
    Batch --> View1[(Batch view)]
    Speed --> View2[(Realtime view)]
    Query1[Query] --> View1
    Query1 --> View2
  end

  subgraph Kappa
    direction LR
    Ev2[Events] --> Stream[Stream processor<br/>Flink / Kafka Streams]
    Stream --> View3[(Materialized view)]
    Query2[Query] --> View3
    Ev2 -. replay .-> Stream
  end

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class Batch,Speed,Stream service;
    class View1,View2,View3 datastore;

Lambda merges fast (approximate) and slow (correct) results. Kappa argues a single stream pipeline with replay can serve both - simpler at the cost of stream-only thinking.

Lambda - two paths, merged at query time#

Nathan Marz introduced Lambda to answer "how do I serve low-latency views without giving up on correctness?" The compromise: process every event twice.

flowchart TB
  subgraph Ingest
    K[(Kafka)]
  end
  subgraph BatchLayer
    K --> HDFS[(HDFS / S3<br/>raw events)]
    HDFS --> Spark[Spark / Hive job]
    Spark --> BV[(Batch view<br/>recomputed daily)]
  end
  subgraph SpeedLayer
    K --> Flink[Flink / Storm]
    Flink --> SV[(Speed view<br/>last few hours)]
  end
  Q[Query API] --> BV
  Q --> SV
  Q -->|merge| Resp[Response]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class Spark,Flink service;
    class K,HDFS,BV,SV datastore;
  • Batch view is correct and immutable. Recomputed from raw events; absorbs late-arriving data and code fixes.
  • Speed view is approximate but fresh. Discarded once the batch view catches up.
  • Query merger picks the freshest correct answer.

Pain points#

  • Two codebases for the same business logic.
  • Two operational paths to monitor.
  • Stitching batch + speed at query time is non-trivial.

Kappa - one stream, replay for reprocessing#

Jay Kreps observed: if your stream processor is exactly-once and your log retains everything, just one pipeline does both jobs. Replay from offset zero to recompute.

flowchart TB
  Ev[Producer] --> K[(Kafka<br/>infinite retention)]
  K --> Flink[Flink / Kafka Streams]
  Flink --> V[(Materialized views<br/>RocksDB-backed)]
  Q[Query] --> V
  Note1[Code change?<br/>Replay from offset 0] -. spawn new job .-> K2[(Kafka)]
  K2 --> Flink2[Flink v2] --> V2[(New view)]
  Q -. cutover .-> V2

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class Flink,Flink2 service;
    class K,K2,V,V2 datastore;
  • One codebase: same DAG used for production and reprocessing.
  • Reprocess via replay: run a parallel job from offset zero, cutover when the new view catches up.
  • Requires exactly-once semantics: idempotent producers + transactional writes.

When to choose which#

Situation Pick
Already on Hadoop, legacy batch jobs Lambda
Greenfield analytics, modern stack Kappa
Batch view requires non-stream operations (full sort, ML training) Lambda
Backfills are rare and small Kappa
Long-term log retention is feasible Kappa
Need precomputed daily aggregates for cost reasons Lambda

Modern hybrids#

Most real systems live between the two:

  • Delta Lake / Iceberg / Hudi: append-only tables on object storage with batch and stream readers.
  • Materialized views in stream processors: Kafka Streams KTable is effectively a Kappa view.
  • Snowflake Streams, BigQuery sub-second streaming: warehouses adding streaming ingest.

The decision is less "Lambda vs Kappa" today and more "how much do I lean toward the streaming end?"

How lambda/kappa relates#

flowchart TB
  LK((Lambda /<br/>Kappa))
  STR[Batch / stream processing<br/>runtime primitives]
  CDC[Change Data Capture<br/>source of events]
  LAKE[Data lake / warehouse<br/>storage tier]
  RTA[Realtime analytics<br/>downstream consumer]
  STR --> LK
  CDC --> LK
  LK --> LAKE
  LK --> RTA

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class LK service;
    class STR,CDC,LAKE,RTA datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Batch stream processing windowing and watermarks batch-stream-processing
HLD Change Data Capture source of stream events from DBs change-data-capture
HLD Data Lake Warehouse the storage layer underneath data-lake-warehouse
HLD Realtime analytics top consumer of Kappa-style pipelines realtime-analytics

Quick reference#

Lambda#

  • Batch layer (Spark/Hive) + Speed layer (Flink/Storm) merged at query.
  • Pros: batch view is correct, recomputable.
  • Cons: two codebases; merge logic non-trivial.

Kappa#

  • Single stream pipeline; replay log to reprocess.
  • Pros: one codebase; simpler ops.
  • Cons: requires exactly-once + infinite log retention.

Choose Lambda when#

  • Legacy Hadoop
  • Batch needs non-stream ops (full sort, ML train)
  • Daily precomputed aggregates for cost

Choose Kappa when#

  • Greenfield
  • Backfills rare and small
  • Storage cheap enough for long retention

Modern hybrids#

  • Delta Lake, Iceberg, Hudi
  • Kafka Streams KTable
  • Snowflake Streams, BigQuery streaming inserts

Refs#

  • Marz - Lambda Architecture blog
  • Kreps - Questioning the Lambda Architecture
  • DDIA ch 11-12

FAQ#

What is Lambda architecture?#

Lambda runs two parallel pipelines: a batch layer that reprocesses all history and a speed layer for sub-minute updates. Queries merge the two views.

What is Kappa architecture?#

Kappa replaces both layers with a single streaming pipeline. To reprocess history, the team replays the event log through the same stream job into a new view.

Lambda vs Kappa: which should I pick?#

Pick Kappa when one stream engine handles both fresh and historic workloads cleanly. Pick Lambda when batch tools (Spark, Hive) give significantly cheaper reprocessing.

What is the main downside of Lambda?#

Two pipelines mean two codebases for the same logic, which drift apart over time and double the operational cost of bug fixes and metric definitions.

Which tools power Kappa pipelines?#

Kafka or Pulsar as the durable log, plus Flink, Kafka Streams, or Spark Structured Streaming as the processor. Materialized views land in Postgres, Druid, or Pinot.

Further reading#