Skip to content

ML Feature Store#

Problem statement (interviewer prompt)

Design a feature store that lets data scientists compute features once and serve them to both batch training jobs and online inference with millisecond reads, while guaranteeing training/serving feature parity.

flowchart TB
  Sources[(Source data:<br/>warehouse, streams)] --> Trans[Feature transformations<br/>SQL / PySpark / Flink]
  Trans --> Offline[(Offline store<br/>Parquet / warehouse)]
  Trans --> Online[(Online store<br/>Redis / DynamoDB)]
  Registry[Feature registry] -.metadata.-> Trans
  Train[Training pipeline] --> Offline
  Infer[Inference service] --> Online

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Trans,Train,Infer service;
    class Registry edge;
    class Sources,Offline,Online datastore;

Eliminates the "trained one feature, served a different one" bug class - the #1 cause of online ML model degradation.

A feature store centralises the transformation of raw data into ML features and serves them consistently to training and inference.

Architecture#

flowchart TB
  subgraph Sources
    DW[(Warehouse)]
    Stream[(Kafka events)]
    OPS[(Operational DB CDC)]
  end
  subgraph Compute
    Batch[Batch transform<br/>Spark / SQL]
    Streaming[Streaming transform<br/>Flink]
  end
  subgraph Stores
    Offline[(Offline: Parquet / Iceberg)]
    Online[(Online: Redis / DynamoDB / Cassandra)]
  end
  subgraph Plane[Control plane]
    Reg[Feature registry]
    Mat[Materializer]
    Mon[Monitoring]
  end
  DW --> Batch --> Offline
  Stream --> Streaming --> Online
  Streaming --> Offline
  Batch --> Online
  OPS --> Streaming
  Reg -.def.-> Batch
  Reg -.def.-> Streaming
  Mat -.schedule.-> Batch
  TrainJob[Training job] --> Offline
  InferSvc[Inference service] --> Online

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Batch,Streaming,Mat,TrainJob,InferSvc compute;
    class Reg,Mon edge;
    class DW,Stream,OPS,Offline,Online datastore;

Feature definition#

A feature lives in the registry as a versioned object:

@feature_view(entities=[user], ttl=timedelta(days=30))
def user_revenue_30d(spark):
    return spark.sql("""
      SELECT user_id, SUM(amount) AS revenue
      FROM orders WHERE created_at > now() - INTERVAL 30 DAY
      GROUP BY user_id
    """)

The same code materialises to both stores; downstream consumers (training, inference) reference the registered name.

Point-in-time join#

The hardest problem in feature stores: a training row labelled with "user converted on day X" must see features as they were on day X (not now).

SELECT lbl.user_id, lbl.target, fv.revenue_30d
FROM labels lbl
ASOF JOIN feature_view fv
  ON lbl.user_id = fv.user_id
  AND fv.event_time <= lbl.event_time

Without this, you leak future information into training and the model collapses in production.

Online store SLOs#

  • Latency: p99 < 10ms per lookup; often hundreds of features per request.
  • Throughput: 100k+ QPS per inference fleet.
  • Freshness: minutes for batch-derived; seconds for streaming-derived.

Redis or DynamoDB for sub-ms; Cassandra for higher throughput at higher latency.

Monitoring#

  • Feature drift: distribution today vs training set.
  • Null rate: feature unexpectedly missing.
  • Freshness lag: time since last update per feature.
  • Skew: difference between online and offline feature values.

When you need a feature store#

Without With
1-2 models, simple features Probably overkill
10+ models sharing user features Worth it
Real-time + batch features both required Yes
Training/serving skew bugs in production Yes

Production references#

Tool Type
Feast OSS, registry-only; bring your own stores
Tecton SaaS, full platform
Vertex AI Feature Store GCP-managed
SageMaker Feature Store AWS-managed
Databricks Feature Store Within Databricks
Hopsworks OSS + commercial

How feature store fits#

flowchart TB
  FS((ML feature<br/>store))
  LK[Lambda / Kappa<br/>dual-store mirror]
  CDC[Change Data Capture<br/>streaming source]
  CACHE[Caching strategies<br/>online store as cache]
  MLI[ML inference pipeline<br/>online consumer]
  LK --> FS
  CDC --> FS
  FS -. uses .- CACHE
  FS --> MLI

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class FS service;
    class LK,CDC,CACHE,MLI datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Lambda Kappa architecture the dual-store mirror lambda-kappa-architecture
HLD Change Data Capture source of streaming features change-data-capture
HLD Caching strategies online store is a cache caching-strategies
HLD ML inference pipeline the downstream consumer ml-inference-pipeline

Quick reference#

Purpose#

One feature definition → consistent values for training (offline) and inference (online).

Components#

  • Registry (definitions)
  • Materializer (schedules)
  • Offline store (Parquet/warehouse)
  • Online store (Redis/Dynamo)
  • Monitoring (drift, freshness, skew)

Point-in-time join#

Training labels at time T see features ≤ T only. ASOF JOIN.

Online SLO#

  • p99 < 10ms
  • 100k+ QPS
  • 100s of features per request

Freshness#

  • Batch (warehouse): minutes-hours
  • Streaming (Flink): seconds

Monitoring metrics#

  • Drift (KL divergence vs training set)
  • Null rate
  • Freshness lag
  • Online/offline skew

Tools#

  • Feast (OSS)
  • Tecton (SaaS)
  • Vertex AI Feature Store
  • SageMaker Feature Store
  • Databricks Feature Store
  • Hopsworks

When worth it#

  • 10+ models sharing features
  • Real-time + batch both
  • Training/serving skew bugs

Refs#

  • feast.dev docs
  • Uber Michelangelo blog
  • Vertex AI Feature Store

FAQ#

How does an ML feature store work?#

Feature pipelines compute features once and write them to both an online store for low-latency serving and an offline store for batch training, with shared definitions guaranteeing parity.

What is training-serving skew?#

Skew happens when features used during training differ from features at inference. A feature store eliminates skew by serving the same transformations to both.

What is point-in-time correctness?#

Point-in-time joins look up the feature value that existed at the moment of the training label, preventing label leakage from future data into the model.

How is the online store different from the offline store?#

The online store is a low-latency KV like Redis or DynamoDB for inference. The offline store is a warehouse or lake like Snowflake or Parquet for batch training and analysis.

How are features kept fresh?#

Streaming pipelines update real-time features in seconds. Batch pipelines refresh slowly changing features on a schedule, and TTLs evict stale entries from the online store.

Why use Feast or Tecton instead of building from scratch?#

Open-source and managed feature stores provide a registry, pipelines, point-in-time joins, and SDK integrations out of the box, saving significant infrastructure work.

Further reading#