Skip to content

LLM Serving Infrastructure#

Problem statement (interviewer prompt)

Design infrastructure to serve a 70B-parameter LLM with token streaming. Handle 10k concurrent users, GPU memory ceiling per replica, p50 first-token latency under 1s, and graceful scaling.

flowchart TB
  C([Client]) --> GW[API Gateway<br/>auth + rate limit]
  GW --> Q[(Request queue)]
  Q --> Sched[Scheduler<br/>continuous batching]
  Sched --> R1[Inference replica<br/>vLLM / TGI on GPU]
  Sched --> R2[Inference replica]
  Sched --> R3[Inference replica]
  R1 --> Stream[(SSE / WebSocket stream)]
  Stream --> C
  Reg[(Model registry)] -.weights.-> R1
  Reg -.weights.-> R2
  Reg -.weights.-> R3

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C client;
    class GW edge;
    class Sched service;
    class R1,R2,R3 compute;
    class Q,Stream queue;
    class Reg datastore;

Key tricks: continuous batching (interleave decode steps from many sessions), paged KV cache (reuse GPU memory for sessions of varying lengths), and tensor / pipeline parallelism (split a model across GPUs that don't fit it on one).

LLM inference is GPU-bound, memory-bound, and bursty. The serving system squeezes throughput out of every megabyte of HBM while keeping per-user latency bounded.

Why LLM serving is special#

Property Effect
Model is huge (10s-100s GB) Doesn't fit on one GPU above 70B params
Autoregressive decode Token N+1 depends on token N; sequential per-session
KV cache grows with context Per-session GPU memory
Highly variable session length Naive batching wastes GPU on padding
Streaming output Per-token latency matters as much as total

Continuous batching#

flowchart LR
  S1[Session A: 'Hello']  --> Step1[Decode step: produce next token for all in batch]
  S2[Session B: 'Write']
  S3[Session C: 'Translate']
  Step1 --> Step2
  S4[New session D] --> Step2
  Step2 --> Step3
  S2 -. completes .-> Out

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Step1,Step2,Step3 compute;
    class Out queue;

Sessions enter and leave the batch every step. A session that finishes mid-batch frees its slot for a new arrival. Boosts GPU utilization 5-10x vs static batching.

Paged KV cache (vLLM)#

Each session needs kv_cache_size = 2 × layers × heads × head_dim × seq_len × dtype_bytes. A 70B model at 8k context is ~10 GB per session.

vLLM stores KV in fixed-size pages (like virtual memory), so a 50-token session uses 50 tokens of cache, not the worst-case allocation. Result: many more concurrent sessions per GPU.

Multi-GPU model parallelism#

flowchart LR
  subgraph Node1[GPU 0 + GPU 1]
    L1[Layer 1 partition] --- L2[Layer 1 partition]
    L1 --- L3[Layer 2 partition]
  end
  subgraph Node2[GPU 2 + GPU 3]
    L4[Layer 3 partition]
    L5[Layer 4 partition]
  end
  Node1 -->|NCCL all-reduce| Node2

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class L1,L2,L3,L4,L5 compute;
  • Tensor parallel: split each layer's matmul across GPUs (heavy NCCL traffic).
  • Pipeline parallel: each GPU owns whole layers; activations flow through (less traffic, more pipeline bubbles).
  • Expert parallel (MoE): route tokens to expert sub-networks on different GPUs.

Quantization#

Precision Memory Quality
FP16 / BF16 2 bytes / param baseline
INT8 1 byte minor loss
INT4 / NF4 / AWQ / GPTQ 0.5 byte small but real loss

70B FP16 = 140 GB; INT4 = 35 GB - fits on a single 40 GB A100 (with overhead) or two 24 GB GPUs.

Speculative decoding#

A small "draft" model proposes K tokens; the big model verifies in one forward pass. If all K are accepted, you got K tokens for the cost of 1 big forward pass.

Streaming#

SSE or WebSocket pushes tokens as they're produced. Time-to-first-token (TTFT) matters more than total time for chat UX.

Autoscaling#

Signal Action
Queue depth > N Add replica
Queue p99 wait > 1s Add replica
GPU utilization > 90% sustained Add replica
Replica cold-start 30-60s to load 70B weights

Cold start is the killer. Tactics: pre-warm pools, snapshot+restore, host-side model cache.

Cost levers#

  • Right-size to model: don't put a 7B on an H100.
  • Quantize aggressively for serving (training stays FP16).
  • Distinguish chat (high TTFT priority) vs batch (high throughput, latency-tolerant).
  • Cache prompts (Anthropic, OpenAI offer this; saves on common system prompts).

Where LLM serving connects#

flowchart TB
  LLM((LLM<br/>serving))
  CO[Container orchestration<br/>runtime]
  GW[API Gateway<br/>entry + rate limit]
  RAG[Vector search / RAG<br/>top consumer]
  RT[Realtime protocols<br/>SSE / WebSocket streaming]
  CO --> LLM
  GW --> LLM
  RAG --> LLM
  LLM --> RT

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class LLM service;
    class CO,GW,RAG,RT datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Container Orchestration the runtime layer container-orchestration
HLD API Gateway the entry point api-gateway
HLD Vector search / RAG top consumer vector-search-rag
HLD Realtime protocols streaming via SSE / WebSocket realtime-protocols

Quick reference#

Why special#

  • 10s-100s GB models, multi-GPU
  • Autoregressive (sequential)
  • KV cache grows with context
  • Variable session length
  • Streaming output

Key tricks#

Trick Effect
Continuous batching Sessions enter/leave per step; 5-10× GPU util
Paged KV cache (vLLM) Pages KV memory; many more concurrent sessions
Tensor parallel Split each layer across GPUs
Pipeline parallel Each GPU owns whole layers
Quantization (INT8/INT4) 70B fits on smaller GPUs
Speculative decoding Small model proposes, big verifies
Prompt caching Reuse system-prompt KV across calls

Capacity math#

  • 70B FP16 = 140 GB; INT4 = 35 GB
  • KV cache 70B 8k ctx ≈ 10 GB per session
  • A100 80GB INT4 70B: ~3 concurrent 8k sessions; many more with shorter contexts via paged

Streaming#

SSE or WebSocket; TTFT (time-to-first-token) is the headline metric.

Autoscaling signals#

Queue depth, p99 wait, GPU util, replica cold-start (30-60s for 70B).

Tooling#

  • vLLM, TGI, TensorRT-LLM, MLC-LLM, llama.cpp
  • Triton Inference Server
  • Ray Serve

Cost levers#

  • Right-size GPU to model
  • INT4 quantize for serving
  • Separate chat (TTFT) vs batch (throughput)
  • Prompt cache for common system prompts

Refs#

  • Kwon et al. - vLLM paper (2023)
  • vllm.ai docs
  • HF TGI docs
  • Anthropic prompt caching

FAQ#

How do you serve a 70B parameter LLM?#

Shard model weights across multiple GPUs (tensor or pipeline parallel), run an inference server like vLLM with continuous batching, stream tokens over SSE, and autoscale replicas on queue depth.

What is continuous batching in LLM serving?#

Instead of waiting to fill a static batch, the scheduler swaps newly arrived requests into idle slots as in-flight requests finish tokens. This raises GPU utilization without hurting latency.

What is the KV cache in LLM inference?#

Each token's attention keys and values are cached so subsequent tokens reuse them instead of recomputing. KV cache memory grows with context length and dominates GPU memory at long contexts.

How do you stream tokens to the client?#

The inference server emits each generated token to an SSE or WebSocket connection. The gateway forwards bytes downstream so users see output incrementally rather than after the full response.

How do you autoscale LLM replicas?#

Track request queue depth, p95 first-token latency, and GPU utilization. Add replicas when queue grows beyond a threshold; preload weights from a model registry to keep cold starts short.

Further reading#