Skip to content

IoT Data Ingestion#

Problem statement (interviewer prompt)

Design an ingestion pipeline for 10M IoT devices publishing telemetry at varying rates (1 msg/s to 1 msg/hour) over MQTT. Handle authentication, partial connectivity, message storage at scale, and real-time alerting on anomalies.

flowchart TB
  D1([Device 1]) -->|MQTT| Br[Broker cluster]
  D2([Device 2]) --> Br
  Dn([Device N]) --> Br
  Br --> Auth[Auth + tenant gate]
  Auth --> Kafka[(Kafka)]
  Kafka --> Stream[Stream processor Flink]
  Stream --> TSDB[(Time-series DB)]
  Stream --> Alert[Alert engine]
  Stream --> Lake[(Data lake parquet)]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class D1,D2,Dn client;
    class Br,Auth edge;
    class Stream,Alert service;
    class Kafka queue;
    class TSDB datastore;
    class Lake storage;

MQTT broker for persistent device connections; Kafka for fan-in to back-end processing; TSDB for queries; lake for retention.

IoT pipelines look like any event-streaming system but with millions of low-power clients on intermittent networks. The constraints reshape every layer.

Device side#

  • Limited compute/memory; lightweight protocols (MQTT, CoAP).
  • Intermittent connectivity; store-and-forward on local flash.
  • Battery: deep sleep between bursts; one bulk publish.
  • Auth via per-device X.509 cert (provisioned at manufacture).

Broker fleet#

flowchart TB
  D1([Devices]) --> LB[L4 LB / DNS]
  LB --> B1[Broker node 1]
  LB --> B2[Broker node 2]
  LB --> Bn[Broker node N]
  B1 <-->|cluster sync| B2
  B2 <-->|cluster sync| Bn
  B1 --> Kafka[(Kafka per-topic)]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class D1 client;
    class LB edge;
    class B1,B2,Bn service;
    class Kafka queue;

MQTT brokers (HiveMQ, EMQX, Mosquitto, AWS IoT Core) hold persistent connections - one per device. A single broker node holds 10k-1M connections; cluster horizontally for more.

QoS levels (MQTT)#

QoS Guarantee
0 At most once (fire and forget)
1 At least once (may duplicate)
2 Exactly once (4-way handshake; expensive)

Most telemetry uses QoS 1.

Auth and provisioning#

sequenceDiagram
  participant Dev as Device
  participant Prov as Provisioning service
  participant CA as Internal CA
  Dev->>Prov: bootstrap token + serial
  Prov->>CA: issue cert(client_id=device-N)
  CA-->>Prov: cert
  Prov-->>Dev: cert + key + broker URL
  Dev->>Broker: mTLS with cert

Per-device certs prevent one compromised device from impersonating others.

Stream processing#

Kafka → Flink / Spark Streaming / Kinesis Analytics:

  • Windowed aggregates (per device, per region, per minute).
  • Anomaly detection (z-score, isolation forest).
  • Real-time alerts.
  • Downsampling for long-term storage.

Storage tiers#

Tier Resolution Retention
Hot (TSDB: InfluxDB, Timescale) Raw 1s 7 days
Warm (TSDB) 1m rollup 90 days
Cold (Parquet / S3) 1h rollup Forever

Device shadow / state#

Many IoT platforms maintain a "device shadow": a JSON document representing desired vs reported state. Updates propagate back to the device on next connect.

Capacity sketch#

10M devices × 1 msg / 10s = 1M msgs/s peak.

  • Broker: 100 nodes × 100k connections each.
  • Kafka: 200 partitions × 5k msgs/s.
  • TSDB: 1M points/s write, compressed.
  • Lake: 1 TB/day raw.

Where IoT ingest fits#

flowchart TB
  IOT((IoT<br/>ingestion))
  PS[Pub/Sub pattern<br/>broker abstraction]
  TSDB[Time-series database<br/>query tier]
  TLS[TLS / mTLS<br/>per-device cert auth]
  STR[Batch / stream processing<br/>analytics layer]
  PS --> IOT
  IOT --> TSDB
  TLS --> IOT
  IOT --> STR

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class IOT service;
    class PS,TSDB,TLS,STR datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Pub Sub Pattern broker abstraction pub-sub-pattern
HLD Time-series Database the query layer time-series-database
HLD Batch stream processing the analytics layer batch-stream-processing
HLD TLS mTLS per-device cert auth tls-mtls

Quick reference#

Constraints#

  • Millions of devices, intermittent connectivity
  • Low compute/battery on devices
  • Per-device auth via X.509
  • Persistent connections for downlink

Components#

  • MQTT broker cluster (HiveMQ, EMQX, AWS IoT Core)
  • Auth gateway (mTLS, JWT)
  • Kafka (fan-in to back end)
  • Stream processor (Flink, Spark, Kinesis Analytics)
  • TSDB (hot/warm tiers)
  • Data lake (cold tier)
  • Device shadow service

MQTT QoS#

  • 0 - fire & forget
  • 1 - at least once (most telemetry)
  • 2 - exactly once (expensive 4-way handshake)

Tiered storage#

Tier Resolution Retention
Hot TSDB 1s 7d
Warm TSDB 1m rollup 90d
Cold lake (Parquet) 1h rollup forever

Capacity sketch#

10M devices × 1 msg/10s = 1M msgs/s peak. Broker: 100 × 100k conns. Kafka: 200 partitions × 5k msgs/s.

Device shadow#

JSON desired vs reported; reconciled on next connect.

Refs#

  • AWS IoT Core docs
  • EMQX docs
  • MQTT 5 spec

FAQ#

How do you ingest data from 10M IoT devices?#

Devices publish over MQTT to a broker cluster sharded by device id range. The broker authenticates, forwards into Kafka, and a stream processor writes to a time-series DB and triggers alerts.

Why MQTT instead of HTTP for IoT?#

MQTT keeps a long-lived TCP connection with tiny 2-byte headers, supports QoS levels, and works on flaky low-bandwidth links. HTTP adds per-request overhead that drains battery.

How are IoT devices authenticated?#

Each device has a unique X.509 certificate provisioned at manufacture. The broker validates the cert against a CA and pins the device to a tenant, so a leaked cert cannot impersonate others.

How do you store IoT time-series data?#

Use a TSDB (InfluxDB, TimescaleDB, Prometheus) for recent hot data and roll older data into a data lake (Parquet on S3). Downsampling keeps long-term queries cheap.

How are real-time alerts generated?#

A stream processor like Flink runs windowed rules and ML models over the Kafka topic. When a rule fires, it writes to an alert topic that a notification service consumes.

Further reading#