Skip to content

A/B Testing Platform#

Problem statement (interviewer prompt)

Design an A/B testing platform: define experiments with variants, assign users deterministically into variants (sticky bucketing), expose flags to client + server SDKs, collect exposure + conversion events, and compute experiment results with proper statistical significance.

flowchart LR
  U([User])
  ASS[Assignment Service]
  EXP[(Experiment Config)]
  EVT[Events Pipeline]
  ANL[Analysis<br/>p-values, lift]
  U --> ASS --> U
  EXP --> ASS
  U --> EVT --> ANL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class U client;
    class ASS,EVT,ANL service;
    class EXP datastore;

flowchart TB
  subgraph Author[Experiment authoring]
    UI([Web UI / yaml definition])
    METR[Metric registry]
    GUARD[Guardrail metrics]
    REVIEW[Review + approval]
  end

  subgraph Cfg[Config plane]
    CFG[(Experiment registry)]
    GIT[Git source of truth]
    PUB([Push to edge / SDK])
  end

  subgraph Assign[Assignment]
    SDK([Client SDK / server])
    HASH[Hash user_id + exp -> bucket]
    OVR[Force overrides]
    EXCL[Mutual exclusion groups]
    CONS[Sticky consistent assignment]
  end

  subgraph Events
    EXP_EVT[Exposure events]
    METR_EVT[Metric events]
    KAFKA[[Kafka]]
    LAKE[Data lake]
  end

  subgraph Analysis
    JOB[Daily / streaming jobs]
    CUPED[CUPED variance reduction]
    SEQ[Sequential testing]
    P[p-values + effects]
    DASH[Dashboard]
    PEEK[Peeking protection]
  end

  subgraph Safety
    GUARDR[Guardrail breaches alert]
    KILL[Kill switch]
    SRM[Sample ratio mismatch detector]
  end

  Author --> Cfg --> Assign
  Assign --> EXP_EVT --> KAFKA --> LAKE --> Analysis
  Safety --- Cfg

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class UI,PUB,SDK client;
    class REVIEW,GIT,OVR,EXCL,CONS,EXP_EVT,LAKE,JOB,CUPED,SEQ,P,PEEK,KILL,SRM service;
    class CFG datastore;
    class KAFKA queue;
    class HASH storage;
    class METR,GUARD,METR_EVT,DASH,GUARDR obs;

Assignment#

  • Deterministic hash of (user_id, experiment_id) → bucket.
  • Mutually exclusive experiments share a salt to avoid overlap.
  • Sticky across sessions; new users get sticky cookie.

Stats#

  • SRM detector flags broken randomization.
  • CUPED reduces variance using pre-experiment metrics.
  • Sequential testing or always-valid p-values avoid early peeking inflation.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD Observability metrics, logs, traces, SLOs observability

Quick reference#

Functional#

  • Define experiment + variants + metrics.
  • Bucket users deterministically.
  • Collect exposures + metric events.
  • Compute lifts with statistical rigor.
  • Guardrails + kill switch.

Non-functional#

  • Assignment latency < 1 ms (client cache).
  • Daily refresh of analyses, with streaming for guardrails.
  • Reliable randomization.

Trade-offs#

  • Server-side vs client-side assignment: server eliminates leakage to client modifying.
  • Fixed-horizon vs sequential: sequential lets you peek safely.
  • Many overlapping experiments demands mutual-exclusion groups.

Refs#

  • "Trustworthy Online Controlled Experiments" Kohavi et al.
  • Microsoft ExP, Booking, LinkedIn, Airbnb experimentation blog posts.
  • Optimizely / GrowthBook docs.

FAQ#

How does an A/B testing platform bucket users?#

Users are hashed by ID modulo a deterministic salt so they consistently fall into the same variant. This sticky bucketing guarantees the same user sees the same experience.

How is statistical significance computed in A/B tests?#

Platforms run a frequentist t-test or chi-square test on conversion metrics, or use Bayesian methods to compute the probability that one variant is better than another.

What is the difference between A/B testing and feature flags?#

Feature flags toggle code paths, while A/B testing measures the causal impact of those toggles using random assignment and statistical analysis of business metrics.

How do experimentation platforms prevent false positives?#

They apply sequential testing corrections, cap peeking, require minimum sample sizes, and pre-register guardrail metrics to detect harmful variants early.

How are experiment exposure events collected?#

Client and server SDKs emit exposure events whenever a variant is shown, joined later with conversion events in a metrics pipeline for per-experiment analysis.

What is a guardrail metric in A/B testing?#

Guardrail metrics like latency or error rate must not degrade in a winning variant. They protect against shipping a treatment that boosts one KPI while hurting another.