Skip to content

Real-time Analytics#

Problem statement (interviewer prompt)

Design a real-time analytics platform for clickstream events: ingest 1M+ events/s, sessionise per user, compute funnel + retention + cohort metrics with sub-minute freshness, and let analysts query both live and historical data with sub-second latency for dashboards.

flowchart LR
  E[Events]
  K[[Kafka]]
  ST[[Stream Processor<br/>Flink / Kinesis]]
  AGG[(Aggregates)]
  OLAP[(OLAP store<br/>Druid / ClickHouse)]
  DASH[Dashboards]
  E --> K --> ST --> AGG --> OLAP --> DASH

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class E service;
    class AGG,OLAP datastore;
    class K,ST queue;
    class DASH obs;

flowchart TB
  subgraph Source
    WEB([Web pixels])
    SDK([Mobile SDK])
    SVC[Server events]
  end

  subgraph Ingest
    LB
    COL([Collector / Edge ingester])
    KAFKA[[Kafka]]
    SCHEMA[Schema registry]
  end

  subgraph Sess[Sessionization & enrich]
    SES[Session windowing]
    JOIN[[Stream-stream / stream-table joins]]
    ENRICH([Geo / device / UTM])
  end

  subgraph Stream
    FLINK[[Flink / Kinesis Analytics]]
    HLL[HLL unique counts]
    TUMBLE[Tumbling windows]
    SLIDE[Sliding windows]
  end

  subgraph Storage
    HOT[(KV / Redis hot counters)]
    OLAP[(Druid / ClickHouse / Pinot)]
    LAKE[(S3 / Iceberg)]
  end

  subgraph Serve
    API[Query API]
    DASH[Dashboards]
    ALERT[Real-time alerts]
    EXPORT[Export to warehouse]
  end

  Source --> LB --> COL --> KAFKA --> SCHEMA
  KAFKA --> Sess --> Stream --> Storage --> Serve

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class WEB,SDK,ENRICH client;
    class SVC,SCHEMA,SES,HLL,TUMBLE,SLIDE,API service;
    class OLAP,EXPORT datastore;
    class HOT cache;
    class KAFKA,JOIN,FLINK queue;
    class COL compute;
    class LAKE storage;
    class DASH,ALERT obs;

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD Probabilistic data structures Bloom, HLL, Count-Min, MinHash, t-digest probabilistic-data-structures
HLD Batch & stream processing Lambda vs Kappa, watermarks, windows batch-stream-processing

Quick reference#

Functional#

  • Sessionization, funnels, retention.
  • Per-event dashboards & ad-hoc queries.
  • Real-time alerts.
  • Export to warehouse for SQL.

Non-functional#

  • Pipeline lag < 5 s.
  • Sub-second dashboard for last 24h.
  • Petabytes long-term in lake.

Trade-offs#

  • Druid/Pinot for high-QPS dashboards; ClickHouse for ad-hoc SQL.
  • Lambda vs Kappa: kappa simpler if stream-first.
  • Exactly-once is expensive; design idempotent counters.

Refs#

  • Druid, Pinot, ClickHouse docs.
  • "Streaming Systems" Tyler Akidau.
  • Snowplow / Heap engineering posts.

FAQ#

How does a real-time analytics platform work?#

Events stream into Kafka, a stream processor like Flink sessionizes and aggregates them, and results land in an OLAP store such as Druid or ClickHouse for sub-second dashboard queries.

What is the difference between Lambda and Kappa architecture?#

Lambda runs parallel batch and streaming pipelines, then reconciles. Kappa uses only a streaming pipeline that can replay history, simplifying ops but demanding robust state.

How is sessionization done at scale?#

Stream processors keyed by user ID maintain session state with an inactivity timeout. When the user goes idle, the session is closed and emitted with computed metrics.

Why use Druid, Pinot, or ClickHouse?#

These OLAP engines store data in columnar segments, pre-aggregate where possible, and use bitmap indexes to answer slice-and-dice queries on billions of rows in sub-second.

How do funnel and cohort analyses work?#

Funnel queries match ordered event sequences per user. Cohort queries group users by an acquisition event and measure retention or revenue over time relative to that anchor.

How is exactly-once guaranteed in streaming analytics?#

Flink and similar engines use distributed snapshots and transactional sinks so a job restart resumes from a consistent state without reprocessing or losing events.