Skip to content

Webhooks System#

Problem statement (interviewer prompt)

Design a webhooks delivery service: publishers register subscribers, when an event fires you POST to every subscriber URL with idempotent retries, exponential backoff, signing for authenticity, a dead-letter queue for failures, and a manual-replay UI.

flowchart LR
  PROD[Producer Service]
  Q[(Outbound queue)]
  DEL([Delivery Workers])
  CUS([Customer Endpoint])
  DLQ[(DLQ)]
  PROD --> Q --> DEL --> CUS
  DEL --> DLQ

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class CUS client;
    class PROD service;
    class Q,DLQ datastore;
    class DEL compute;

flowchart TB
  subgraph Producers
    APP[App services]
    OUTBOX[[Outbox events]]
  end

  subgraph Pipeline
    BUS[[Event bus]]
    SUB([Subscription registry<br/>customer endpoints])
    FILTER[Event filter / type]
    TRANS[Payload templater + transform]
  end

  subgraph Delivery
    POOL([Delivery worker pool])
    SIGN[HMAC signing]
    RETRY[Exponential backoff + jitter]
    BUDGET[Per-endpoint retry budget]
    CB[Circuit breaker per endpoint]
    DLQ[(DLQ)]
    REPLAY[Manual replay UI]
  end

  subgraph Customer
    EP([Customer endpoint])
    IDEM([Customer idempotency])
  end

  subgraph Ops
    OBS[Metrics: delivered, retry, latency]
    AUDIT[Audit log]
    SECRETS([Per-customer signing secret])
  end

  Producers --> BUS --> FILTER --> TRANS --> POOL --> EP
  SUB --- FILTER
  POOL --> RETRY --> POOL
  POOL --> CB
  POOL --> DLQ --> REPLAY
  POOL --> SIGN
  SECRETS --- SIGN
  Ops --- POOL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class SUB,EP,IDEM,SECRETS client;
    class APP,FILTER,TRANS,SIGN,RETRY,BUDGET,CB,REPLAY service;
    class DLQ datastore;
    class OUTBOX,BUS queue;
    class POOL compute;
    class OBS,AUDIT obs;

Delivery semantics#

  • At-least-once with HMAC-signed bodies and Idempotency-Key header.
  • Customer side: dedupe by (event_id, type).

Retry policy#

  • 5xx / timeout → backoff sequence (1s, 2s, 4s, … up to days).
  • 4xx (auth, validation) → don't retry; alert customer.
  • Per-endpoint circuit breaker: open after high failure rate to protect the system.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD Idempotency & retries safe re-execution, backoff + jitter idempotency-retries
HLD Resilience patterns timeout, retry, breaker, bulkhead, backpressure resilience-patterns
HLD Observability metrics, logs, traces, SLOs observability
LLD REST API design verbs, statuses, pagination, errors rest-api-design

Quick reference#

Functional#

  • Register endpoints + subscribed event types.
  • Sign payloads (HMAC).
  • Retry with backoff.
  • DLQ + manual replay.
  • Per-endpoint health.

Non-functional#

  • Delivery within seconds normally; retries can stretch days.
  • 99.9% eventual delivery.

Trade-offs#

  • Fanout architecture matters at scale (one event → many endpoints).
  • Customer endpoints flake: design for them.
  • Per-endpoint isolation so one slow customer doesn't block others.

Refs#

  • Stripe webhook architecture blog.
  • "Convoys" (avoiding head-of-line blocking) Stripe blog.
  • Svix, hookdeck open-source webhook gateways.

FAQ#

How do webhooks work?#

A producer registers an event subscription with a target URL. When the event fires, the webhook service POSTs a signed JSON payload to that URL and retries with exponential backoff on failure.

How do you ensure reliable webhook delivery?#

Persist every event to a durable queue before attempting delivery, retry with exponential backoff for non-2xx responses, and move events that exhaust retries to a dead-letter queue for replay.

How do you verify webhook authenticity?#

The sender signs the payload with an HMAC using a shared secret and sets it in a header. The receiver recomputes the HMAC and compares in constant time to confirm the message is genuine.

What is a webhook dead letter queue?#

The DLQ holds events that failed all delivery retries. Operators can inspect, fix the receiver, and replay them from a UI without losing data.

Webhooks vs polling: which is better?#

Webhooks push events instantly and use far less bandwidth, but require the receiver to expose a public endpoint. Polling is simpler but adds latency and wastes calls on empty intervals.

How do you handle webhook idempotency?#

Include a unique event_id in every delivery and require receivers to dedupe by it. The sender retries safely because duplicate processing is a no-op on the receiver side.