Resilience Patterns#

Problem statement (interviewer prompt)

Walk through the resilience toolkit (timeouts, retries with jitter, circuit breakers, bulkheads, hedged requests, backpressure, graceful degradation). For a checkout flow with five downstream calls, explain which patterns you'd apply at each call and why.

flowchart LR
  C[Caller]
  TO[Timeout]
  RT[Retry + backoff]
  CB[Circuit Breaker]
  BH[Bulkhead<br/>isolated pool]
  FB[Fallback]
  S[Downstream]
  C --> TO --> RT --> CB --> BH --> S
  CB -. open .-> FB

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C,TO,RT,CB,BH,FB,S service;

flowchart TB
  subgraph Stability[Stability patterns]
    TO[Timeout<br/>connect + read]
    RT[Retry with backoff + jitter]
    CB[Circuit Breaker<br/>closed -> open -> half-open]
    BH[Bulkhead<br/>per-dependency pools]
    SH[[Shed Load<br/>queue length / cpu]]
    RL[Rate Limit / Quota]
    HD[Hedged Requests]
    DG[Graceful Degradation]
    FB[Fallback / cached / static]
  end

  subgraph Pressure[Backpressure]
    QS[[Bounded queues]]
    REJ[Reject on full<br/>fail fast]
    BP[Reactive backpressure<br/>RxJava / Reactive Streams]
    FLOW[gRPC / HTTP2 flow control]
  end

  subgraph Failure[Failure injection]
    CHAOS[Chaos Monkey / GameDays]
    DELAY[Latency injection]
    NETWORK[Network partition]
    CRASH[Process kill]
  end

  subgraph Detect[Health & detection]
    LIVE[Liveness probe]
    READY[Readiness probe]
    HC[Active health checks]
    HK[Heartbeats]
    PHI[Phi accrual]
    CIRC[Circuit metrics]
  end

  subgraph Recover[Recovery]
    REST[Restart / kill switch]
    FAIL[Failover replicas]
    SPILL[Spillover region]
    REPLAY[[Replay from log / queue]]
  end

  Caller --> TO --> RT --> CB --> BH --> Downstream
  CB --> FB
  SH --> REJ
  RL --> SH
  HD --> Downstream
  QS --> REJ
  CHAOS -.exercise.-> Detect
  Detect --> Recover

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class TO,RT,CB,BH,RL,HD,DG,FB,REJ,BP,FLOW,CHAOS,DELAY,NETWORK,CRASH,LIVE,READY,HC,HK,PHI,REST,FAIL,SPILL service;
    class SH,QS,REPLAY queue;
    class CIRC obs;

Circuit breaker states#

Closed: requests flow; failures counted in rolling window.
Open: requests rejected immediately; downstream is shielded.
Half-open: after timeout, allow N probe requests; if pass → closed, else → open.

Thresholds: e.g., 50% failure rate over 20 requests in 10 s → open for 30 s.

Bulkhead#

Separate thread/connection pool per dependency.
A slow analytics API can't starve payment calls of threads.

Timeouts#

Always set both connect and read timeouts.
Total timeout for a request = t. If you do N retries, per-attempt timeout ≤ t / (1 + N).
Tail latency dominates - set timeouts at p99 of healthy, not avg.

Hedged requests#

After p95, send a second copy; cancel slower.
Used by Google (BigTable), AWS (S3 hot reads).
Risk: amplifies load → use sparingly with budget.

Backpressure#

Bounded queues are mandatory; unbounded = OOM time bomb.
Better to reject (429) than queue forever.
Reactive Streams spec, gRPC flow control, Kafka consumer poll cadence.

Graceful degradation examples#

Cache-only mode if DB down.
Serve stale recommendations if model server down.
Disable comments if comment service down (rather than fail page).

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
`HLD`	Leader/follower replication	sync/semi-sync/async replication, failover	replication-leader-follower
`HLD`	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
`HLD`	Resilience patterns	timeout, retry, breaker, bulkhead, backpressure	resilience-patterns
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`HLD`	HTTP / TLS protocols	HTTP 1.1/2/3, QUIC, TLS 1.3	http-protocols
`LLD`	REST API design	verbs, statuses, pagination, errors	rest-api-design
`LLD`	Async models	futures / async-await / coroutines / actors	async-models

Quick reference#

Default playbook for any service call#

Timeout (connect + read), short.
Retry only idempotent ops, with exponential backoff + jitter, max N.
Circuit breaker around the call.
Bulkhead the pool.
Fallback / static response if all else fails.
Surface SLO impact via metrics.

Sizing concurrency#

Little's Law: concurrency = throughput × latency.
If p99 = 200 ms and target 1000 RPS, you need 200 concurrent slots.
Pool size bigger than that wastes; smaller queues requests.

Failure detection#

Liveness ≠ readiness. Liveness: "process alive?". Readiness: "can serve traffic?".
Heartbeat with phi-accrual: probabilistic measure of suspicion vs binary up/down.

Anti-patterns#

Infinite retry loops without budget.
Long timeouts that hold threads (default 30 s on HTTP libs is too high).
No upstream backpressure → memory blow up.
Circuit breaker per host instead of per dependency.
Retrying inside retrying (nested retry storms).

Where this shows up in this repo#

All service-to-service calls.
API gateway timeouts & circuits.
Job scheduler retry policies.
Webhook delivery retries.
Email/SMS provider failover.
Message queue consumer DLQ.

Refs#

Michael Nygard: "Release It!" (Bulkhead, Circuit Breaker patterns).
Marc Brooker AWS: "What's a 'reasonable' timeout?", "Hedging your bets."
Netflix Hystrix (now archived) docs; Resilience4j docs.
Google SRE Book ch.22 (Addressing Cascading Failures).

FAQ#

What are the three states of a circuit breaker?#

Closed allows traffic through, open short-circuits calls after a failure threshold, and half-open lets a few probes through to test recovery before fully closing again.

Why add jitter to retries?#

Without jitter, many clients retry at the same instant after a shared backoff and pile back onto the failing service. Random jitter spreads the retry storm out over time.

When should I use the bulkhead pattern?#

Use bulkheads when one slow dependency can exhaust thread pools or connections shared with healthy paths. Isolating resources per dependency prevents one bad call from sinking the whole service.

Are retries always safe?#

Only for idempotent operations. Retrying a payment charge or a non-idempotent POST without an idempotency key can double-charge or duplicate writes.

What is a hedged request?#

A hedged request sends a second copy of the same call to a different replica after a short delay and takes whichever response arrives first. It cuts tail latency at the cost of extra load.

API Gateway: API gateways implement circuit breakers, rate limiting, and retry policies as resilience mechanisms
Observability: observability data (metrics, traces) is required to trigger and tune resilience patterns correctly
Distributed Transactions: resilience patterns like sagas and idempotent retries are essential for fault-tolerant distributed transactions