Skip to content

Circuit Breaker#

The circuit breaker pattern protects a caller from a failing downstream by tripping fast when error rates spike. Instead of letting every request time out, the breaker fails immediately for a cool-down window, then probes the dependency to see if it has recovered.

Circuit breaker pattern state diagram: closed, open, and half-open states
Source: Wikimedia Commons. CC BY-SA.
stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failure threshold exceeded
  Open --> HalfOpen: cool-down elapsed
  HalfOpen --> Closed: probe succeeds
  HalfOpen --> Open: probe fails

Without a breaker, a slow dependency can take down its caller. Threads pile up on hanging socket reads, queues fill, and eventually the caller's own consumers time out and retry, multiplying load on an already sick downstream. The classic outage pattern is a service that is only 3 percent unhealthy but takes the whole fleet down because retries from upstream callers turn into a thundering herd.

A circuit breaker wraps every call. In the closed state calls flow normally, but the breaker counts failures (or measures latency) over a sliding window. When a threshold is crossed (say, 50 percent errors over the last 20 calls) the breaker moves to open: subsequent calls fail instantly with no network attempt. After a cool-down (typically a few seconds) it moves to half-open and lets one or a few probe calls through. If they succeed it closes again; if they fail it reopens for another cool-down. Mature libraries (Resilience4j, Polly, Sentinel, Istio outlier detection) add per-endpoint breakers, bulkheading, and metric hooks.

Problem statement (interviewer prompt)

Design call protection for a service that depends on a flaky downstream API so that, when the downstream is slow or failing, the caller fails fast, releases threads, and automatically recovers without manual intervention.

Why circuit breakers exist#

A slow dependency is more dangerous than a dead one. Dead returns an error in microseconds; slow holds the calling thread for seconds while connections, file descriptors, and queue slots leak. Released by Netflix as Hystrix in 2012, the circuit breaker prevents this cascade by detecting failure and short-circuiting calls until the dependency recovers.

flowchart LR
  CLIENT([Caller])
  CB[Circuit Breaker]
  DEP[Downstream API]
  FB[Fallback]

  CLIENT --> CB
  CB -- closed --> DEP
  CB -- open --> FB
  CB -- half-open probe --> DEP

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class CLIENT client;
    class CB edge;
    class DEP external;
    class FB service;

The three states#

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: threshold crossed
  Open --> HalfOpen: cool-down expired
  HalfOpen --> Closed: probes succeed
  HalfOpen --> Open: probe fails
  Closed --> Closed: success or error below threshold
State Behavior Counts
Closed Calls pass through Track failures in sliding window
Open Fail fast, no network call Wait for cool-down timer
Half-open Allow N probe calls only Success closes, failure reopens

Threshold metrics#

Hystrix used a simple error count window. Resilience4j adds richer modes:

Mode Trigger
Count-based Last N calls failure rate above X percent
Time-based Failures per second above X
Slow-call ratio Calls slower than threshold above X percent

A combined rule (e.g. 50 percent error or 50 percent slow over 20 calls) catches both crashes and brownouts. Always require a minimum number of calls before tripping so a single failure on a quiet endpoint does not open the breaker.

Implementation sketch#

from enum import Enum
from time import monotonic

class State(Enum):
    CLOSED = 1
    OPEN = 2
    HALF_OPEN = 3

class CircuitBreaker:
    def __init__(self, fail_threshold=0.5, window=20, cool_down=5.0):
        self.state = State.CLOSED
        self.window = window
        self.fail_threshold = fail_threshold
        self.cool_down = cool_down
        self.results = []  # 1 success, 0 failure
        self.opened_at = 0

    def call(self, fn, *args, **kw):
        if self.state == State.OPEN:
            if monotonic() - self.opened_at >= self.cool_down:
                self.state = State.HALF_OPEN
            else:
                raise CircuitOpenError()
        try:
            result = fn(*args, **kw)
            self._on_success()
            return result
        except Exception:
            self._on_failure()
            raise

    def _on_success(self):
        if self.state == State.HALF_OPEN:
            self.state = State.CLOSED
            self.results.clear()
        self._record(1)

    def _on_failure(self):
        if self.state == State.HALF_OPEN:
            self._trip()
        else:
            self._record(0)
            failure_rate = 1 - (sum(self.results) / len(self.results))
            if len(self.results) >= self.window and failure_rate >= self.fail_threshold:
                self._trip()

    def _record(self, v):
        self.results.append(v)
        if len(self.results) > self.window:
            self.results.pop(0)

    def _trip(self):
        self.state = State.OPEN
        self.opened_at = monotonic()

Hystrix vs Resilience4j vs service mesh#

Library Style Notes
Hystrix Thread pool isolation + breaker Netflix, in maintenance since 2018
Resilience4j Functional, lightweight, modular Modern JVM default
Polly .NET equivalent Used widely on Azure
Sentinel Alibaba, flow control + breakers Cloud-native, dashboard
Istio / Envoy L7 outlier detection Mesh-level, no code change
AWS App Mesh Same idea at the proxy Provider-managed

Service-mesh breakers operate per upstream host; library breakers operate per logical dependency. Both can coexist.

Circuit breaker vs retry with backoff#

These solve different problems and combine well, with care.

Tool Solves Risk if alone
Retry + backoff Transient errors Multiplies load on a sick dep
Circuit breaker Sustained failure Slow to detect intermittent failures
Bulkhead Resource isolation Does not stop hanging threads
Timeout Bounded waits Does not detect repeated failures

Best practice: retry only inside a circuit breaker; the breaker counts the final outcome, not each retry, otherwise one slow call inflates the failure count.

Fallbacks#

A breaker that just throws helps the caller but not the user. Common fallbacks:

  • Cached or stale response (last good value)
  • Default safe value (empty list, zero count)
  • Degraded path (skip personalization, show generic feed)
  • Queue the request for async retry

Fallbacks must be cheap and side-effect-free; a fallback that calls another flaky service defeats the purpose.

Tuning checklist#

  • Per-endpoint breakers, not one global breaker
  • Minimum call count before tripping (avoid noise)
  • Sliding window 10 to 60 seconds typical
  • Cool-down 1 to 10 seconds, exponential on repeat trips
  • Single-flight probes in half-open, not a flood
  • Emit metrics: state, failure rate, slow-call rate, trip count
  • Distinct counters for client errors (4xx) and server errors (5xx); 4xx should usually not trip

Production gotchas#

  • Breakers and load balancers must agree on what counts as failure (status codes, timeouts, RST connections).
  • Do not trip on validation 4xx; those are caller bugs, not downstream failures.
  • Watch for breaker oscillation: a noisy downstream that flips Open/HalfOpen every few seconds wastes more capacity than constant retries.
  • Test the breaker in chaos drills; many teams have breakers configured but never observe them trip.
  • Coordinate with the rate limiter: tripping when the limiter is rejecting calls just adds confusion.
  • Mesh-level breakers can hide application-level breakers; pick one layer or document the interplay.

Quick reference#

States#

  • Closed: pass through, count failures
  • Open: fail fast, no call attempted
  • Half-open: limited probes decide next state

Triggers#

  • Count-based: failure rate over last N calls
  • Time-based: failures per second
  • Slow-call ratio: latency above threshold
  • Require minimum call count before tripping

Defaults that work#

  • Window: last 20 calls or 10 seconds
  • Fail threshold: 50 percent
  • Cool-down: 5 seconds, exponential on repeat trips
  • Half-open probes: 1 to 3 single-flight calls

Compose with#

  • Timeout: bound every call
  • Retry + backoff: only inside the breaker, count final outcome
  • Bulkhead: limit concurrent calls per dependency
  • Fallback: cached value, default, degraded path

Libraries and infra#

  • Resilience4j (JVM, current standard)
  • Polly (.NET)
  • Sentinel (Alibaba)
  • Hystrix (Netflix, maintenance mode)
  • Envoy / Istio outlier detection
  • AWS App Mesh

Failure modes#

  • Tripping on 4xx (caller bugs, not downstream)
  • Flood of probes in half-open
  • Breaker oscillation on noisy upstream
  • Per-instance vs per-service confusion
  • Fallback that calls another flaky dependency

Metrics to emit#

  • Current state
  • Failure rate, slow-call rate
  • Trip count and last trip time
  • Calls allowed, rejected, succeeded

When to use#

  • Any synchronous call to a separately-deployed service
  • External APIs with variable reliability
  • Database or cache calls that can hang

When to skip#

  • Pure in-process calls
  • Async fire-and-forget with retry on the queue
  • Very low traffic where the window never fills

Refs#

  • Michael Nygard: "Release It!" (2007, 2nd ed 2018)
  • Netflix Tech Blog: "Introducing Hystrix" (2012)
  • Resilience4j docs: "CircuitBreaker" (current)

FAQ#

What is the circuit breaker pattern?#

Circuit breaker is a resilience pattern that monitors failures and trips open when error rate crosses a threshold. While open it fails calls fast instead of waiting on timeouts, freeing threads.

What are the three circuit breaker states?#

Closed lets traffic flow while counting errors. Open rejects calls immediately for a cool-down. Half-open lets a probe request through, closing the breaker on success or reopening on failure.

Circuit breaker vs retry, when to use each?#

Retry handles transient failures like a one-off timeout. Circuit breaker handles sustained downstream failure. Combine them: retry inside the closed state, stop retrying once the breaker is open.

How do I tune a circuit breaker?#

Set the failure threshold above natural noise (often 50 percent over a sliding window of 20 to 50 calls), cool-down between 5 and 30 seconds, and slow-call threshold around the SLA limit.

Which libraries implement circuit breaker?#

Resilience4j is the modern JVM standard. Polly is the .NET equivalent. Sentinel is popular in China. Istio offers outlier detection at the proxy level, no app code required.