Skip to content

Data Lake, Warehouse, Lakehouse#

Problem statement (interviewer prompt)

An analytics platform holds petabytes of raw clickstream, processed user metrics, and curated business tables for executive dashboards. Compare the storage architectures (lake, warehouse, lakehouse) and pick the right home for each dataset.

Three reference architectures for analytical storage:

flowchart LR
  subgraph Lake[Data Lake]
    L1[S3 / HDFS] --- L2[Parquet / JSON / images]
    L3[Schema on read]
  end
  subgraph Warehouse[Data Warehouse]
    W1[Snowflake / BigQuery] --- W2[Columnar tables]
    W3[Schema on write]
  end
  subgraph Lakehouse[Lakehouse]
    H1[Delta Lake / Iceberg / Hudi]
    H2[ACID on object storage]
    H3[SQL + ML in one place]
  end

    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    class L1,W1,H1 datastore;
    class L2 storage;

Lakes hold raw bytes cheaply. Warehouses hold curated tables fast. Lakehouses try to do both with table formats (Delta, Iceberg, Hudi) layered over object storage.

Three storage tiers for analytical data, each with its own contract on cost, schema, and concurrency.

Data Lake#

Object storage (S3, GCS, ADLS) holding raw and semi-processed files: JSON logs, Parquet, ORC, images, audio.

  • Schema on read: the analyst defines structure at query time.
  • Cost: cheapest per TB. Object storage at ~$23/TB/month.
  • Engines: Athena, Trino, Spark, Presto.
  • Pain: no transactions, no time travel, no DELETE/UPDATE. Small file proliferation.

Data Warehouse#

Specialised columnar database (Snowflake, BigQuery, Redshift, Synapse) for SQL analytics.

  • Schema on write: defined upfront.
  • Cost: 5-10x more expensive per TB than object storage, but compute is on-demand and serverless.
  • Concurrency: thousands of analysts.
  • Strengths: ACID, time travel, ML extensions, governance.
flowchart TB
  ELT[ELT pipeline<br/>Fivetran / dbt] --> W[(Warehouse)]
  W --> BI[BI tools<br/>Looker, Tableau]
  W --> Notebooks[Jupyter, Mode]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class ELT,BI,Notebooks service;
    class W datastore;

Lakehouse - the convergence#

A lakehouse stores data in open Parquet on object storage but adds a transactional table format:

Table format Owner ACID Time travel Hidden partitioning
Delta Lake Databricks yes yes yes
Iceberg Apache (Netflix-born) yes yes yes
Hudi Apache (Uber-born) yes yes yes 
flowchart TB
  subgraph Storage[Object storage]
    P[Parquet data files] -.metadata.-> M[Transaction log<br/>JSON / manifests]
  end
  subgraph Engines
    Spark
    Trino
    Snowflake
    Flink
  end
  Engines -->|read/write via format spec| Storage

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    class Spark,Trino,Snowflake,Flink service;
    class M datastore;
    class P storage;

Multiple engines can read and write the same dataset safely. Snowflake, BigQuery, and Databricks all support external tables on Iceberg now.

Medallion architecture#

A common layering inside a lake/lakehouse:

flowchart LR
  R[Raw / Bronze<br/>append-only] --> S[Silver<br/>cleaned, conformed] --> G[Gold<br/>aggregated, BI-ready]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
  • Bronze: every event as-received, immutable. Source of truth.
  • Silver: validated, deduped, joined to lookup tables.
  • Gold: business-facing tables; what analysts and dashboards see.

Choosing the right tier#

Dataset Home
Raw clickstream Lake (bronze)
Conformed user dimensions Lakehouse silver
Daily revenue dashboard Warehouse or lakehouse gold
ML training set Lakehouse silver (Parquet for Spark)
Real-time aggregates Stream-state store + lakehouse
Sensitive PII Warehouse with column masking

Cost model in one line#

total = storage * GB-months + compute * scan-bytes * query-count + egress

Optimisations: partitioning, file compaction, ZORDER, projection pushdown, cached aggregates.

Where the data tier sits#

flowchart TB
  DL((Data lake /<br/>warehouse / lakehouse))
  STR[Batch / stream processing<br/>reads + writes tiers]
  OBJ[Object storage<br/>byte substrate]
  CDC[Change Data Capture<br/>bronze ingestion]
  LK[Lambda / Kappa<br/>processing model on top]
  STR --> DL
  CDC --> DL
  DL --> OBJ
  LK --> DL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class DL service;
    class STR,OBJ,CDC,LK datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Lambda Kappa architecture the processing model on top lambda-kappa-architecture
HLD Batch stream processing how data flows into the lake batch-stream-processing
HLD Object storage the substrate of every lake object-storage
HLD Change Data Capture how operational data lands in bronze change-data-capture

Quick reference#

Comparison#

Tier Schema Cost/TB ACID Engines
Lake on-read $$ no Trino, Spark
Warehouse on-write $$$$ yes Snowflake, BigQuery
Lakehouse on-write (open format) $$ yes Spark, Trino, Snowflake

Table formats#

Format Origin
Delta Lake Databricks
Iceberg Netflix → Apache
Hudi Uber → Apache

All three give: ACID, time travel, hidden partitioning, schema evolution.

Medallion layout#

  • Bronze: raw, immutable
  • Silver: cleaned, deduped, conformed
  • Gold: aggregated, BI-ready

Routing table#

Dataset Home
Clickstream raw Bronze (lake)
User dim Silver (lakehouse)
Daily KPI Gold (warehouse)
ML training Silver Parquet
PII Warehouse with masking

Cost#

storage + compute(scan-bytes × queries) + egress
Optimise via partitioning, ZORDER, projection pushdown.

Refs#

  • Armbrust et al. - Lakehouse paper (CIDR 2021)
  • Delta Lake / Iceberg / Hudi docs

FAQ#

What is the difference between a data lake and a data warehouse?#

A data warehouse stores curated structured tables on columnar storage with schema-on-write. A data lake stores raw files in object storage with schema-on-read, accepting any format including images and logs.

What is a lakehouse?#

A lakehouse combines the openness of a data lake with warehouse-grade ACID transactions, schema enforcement, and BI performance, using table formats like Delta Lake, Apache Iceberg, or Hudi on object storage.

Delta Lake vs Iceberg vs Hudi?#

Delta is best on Databricks and tightly integrated with Spark. Iceberg has the broadest engine support (Spark, Trino, Flink, Snowflake) and the cleanest spec. Hudi excels at streaming upserts.

When should I use a lake vs a warehouse?#

Use a warehouse for curated business reporting where SQL latency matters. Use a lake for raw ingestion, ML training, and unstructured data. Use a lakehouse when you want one tier for both.

Snowflake vs Databricks for analytics?#

Snowflake is best for pure SQL BI with zero ops and strong governance. Databricks excels for ML and streaming on open formats. Pick by primary workload, both can do the other.

Further reading#