Skip to content

ML Inference Pipeline#

Problem statement (interviewer prompt)

Design a real-time scoring service: REST endpoint receives a payload, fetches features from a feature store, batches with concurrent requests, scores with a tree or NN model, and returns predictions under 100ms p99.

flowchart TB
  C([Client]) --> GW[API Gateway]
  GW --> Svc[Inference service]
  Svc --> FS[(Feature store online)]
  Svc --> Batch[Micro-batcher<br/>10-50ms windows]
  Batch --> Runner[Model runner<br/>Triton / ONNX / TF Serving]
  Runner --> Reg[(Model registry)]
  Runner --> Post[Post-process]
  Post --> Svc
  Svc --> Log[(Prediction log)]
  Log --> DW[(Warehouse: A/B + drift)]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C client;
    class GW edge;
    class Svc,Batch,Post service;
    class Runner compute;
    class FS,Reg,Log,DW datastore;
Convolutional neural network feature layers visualization
Source: Wikimedia Commons. CC BY-SA / public domain.

Hot paths: feature fetch (cached), micro-batch to maximize GPU throughput, log everything for online evaluation.

The system that turns trained model + live request into a prediction, while remaining fast, observable, and ready to roll back.

Components#

flowchart TB
  C([Client request]) --> GW[API Gateway]
  GW --> Router[A/B router]
  Router --> Svc[Inference service v1]
  Router --> Svc2[Inference service v2 canary]
  Svc --> Cache[(Feature cache local)]
  Cache -. miss .-> FS[(Feature store online)]
  Svc --> MB[Micro-batcher]
  MB --> Runner[Model runner]
  Runner --> Reg[(Model registry)]
  Runner --> Post[Post-process]
  Post --> Svc
  Svc --> Resp([Response])
  Svc --> Log[(Prediction log Kafka)]
  Log --> DW[(Warehouse)]
  Log --> Drift[Drift detector]
  Drift --> Alert[Alerting]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C,Resp client;
    class GW,Router edge;
    class Svc,Svc2,MB,Post,Drift,Alert service;
    class Runner compute;
    class Cache cache;
    class FS,Reg,DW datastore;
    class Log queue;

Latency budget (100ms p99)#

Stage Budget
Network ingress + auth 5ms
Feature fetch (cache hit) 5ms
Feature fetch (cache miss) 20ms
Micro-batch wait 10-30ms
Model forward (CPU model) 5ms
Model forward (GPU NN) 30ms
Post-process + response 5ms

Squeeze cache hit rate above 95%; batch aggressively for GPU models.

Model registry#

Every model: id, version, training data hash, metrics, artifact URI. The inference service pins to a registry version; deploys are atomic and reversible.

A/B and canary#

sequenceDiagram
  participant C as Client
  participant R as Router
  participant V1 as Model v1
  participant V2 as Model v2 canary
  C->>R: request
  alt 95% traffic
    R->>V1: score
    V1-->>C: prediction
  else 5% traffic
    R->>V2: score
    V2-->>C: prediction
  end

Track both arms; promote v2 when business metric proves out.

Drift monitoring#

Signal What it means
Input feature distribution shift Upstream changed; retrain
Prediction distribution shift Model output drifting; investigate
Outcome distribution shift World changed; model stale
p99 latency spike Often a memory pressure

Autoscaling#

  • Scale on QPS / queue depth / p99.
  • Pre-warm pool to dodge cold-start (model load 5-30s).
  • GPU pods: scale on GPU utilization, not just CPU.

Multi-tenant isolation#

  • Per-tenant rate limit at gateway.
  • Per-tenant model variants when needed.
  • Per-tenant prediction logs (compliance).

How inference pipeline fits#

flowchart TB
  MLI((ML inference<br/>pipeline))
  FS[ML feature store<br/>feature source]
  DS[Deployment strategies<br/>canary / shadow]
  OBS[Observability<br/>drift + latency tracking]
  BP[Backpressure<br/>micro-batching]
  FS --> MLI
  DS --> MLI
  MLI --> OBS
  BP -. inside .- MLI

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class MLI service;
    class FS,DS,OBS,BP datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD ML feature store feature source ml-feature-store
HLD Deployment strategies canary / A/B deployment-strategies
HLD Observability drift and latency tracking observability
HLD Backpressure micro-batching is a form backpressure

Quick reference#

Stages#

ingress → auth → feature fetch → micro-batch → model forward → post-process → response + log.

Latency budget (100ms p99)#

  • Auth: 5ms
  • Feature fetch hit: 5ms
  • Batch wait: 10-30ms
  • Forward (CPU): 5ms; (GPU NN): 30ms
  • Post-process: 5ms

Components#

  • Model registry (versioned artifacts + metadata)
  • A/B router
  • Inference service
  • Model runner (Triton, TF Serving, ONNX, vLLM)
  • Prediction log (Kafka → warehouse)
  • Drift detector

Drift signals#

  • Feature distribution shift
  • Prediction distribution shift
  • Outcome shift
  • p99 latency spike

Autoscaling#

  • QPS / queue depth / p99 SLO
  • GPU pods: GPU util signal
  • Pre-warm to dodge 5-30s cold-start

Multi-tenant#

  • Per-tenant rate limit
  • Per-tenant model variants if needed
  • Per-tenant logs for compliance

Tools#

NVIDIA Triton, KServe, BentoML, TorchServe, TF Serving, MLflow, Seldon, Ray Serve.

Refs#

  • NVIDIA Triton docs
  • KServe project
  • Uber - Scaling Michelangelo

FAQ#

How does a real-time ML inference pipeline work?#

A request hits an inference service that fetches features from a feature store, runs the model with optional batching, and returns predictions with sub-100ms latency.

What is dynamic batching in inference?#

The server collects concurrent requests for a short window and runs them as a batch through the model, dramatically improving GPU throughput with small latency cost.

How are ML models versioned in production?#

Each trained model gets a unique version. The serving stack supports canarying new versions against a champion, with traffic splits and automatic rollback on metric regression.

When should you use GPU vs CPU for inference?#

GPUs win for deep neural nets with large matrix multiplies. CPUs are cheaper and lower latency for small models, tree ensembles, and quantized transformers.

How do inference services autoscale?#

Pods scale on CPU, GPU utilization, or queue depth. Pre-warming and model caching avoid cold starts when traffic spikes faster than autoscaler can spin up replicas.

How are per-tenant rate limits enforced?#

The gateway applies token-bucket limits per tenant API key, queues bursts, and rejects overage. This isolates noisy tenants from impacting shared model capacity.

Further reading#