Skip to content

Bulkhead Pattern#

The bulkhead pattern fault isolation idea comes from ships: a hull is divided into watertight compartments so a single hole sinks one section, not the whole vessel. In services, a "bulkhead" is a dedicated pool of resources (threads, connections, semaphores) per dependency or per tenant. If recommendations hangs, it cannot drain the threads that payments needs.

flowchart LR
  C[Client requests]
  subgraph App[Service process]
    direction TB
    P1[Pool A<br/>payments<br/>20 threads]
    P2[Pool B<br/>search<br/>30 threads]
    P3[Pool C<br/>analytics<br/>5 threads]
  end
  D1[(Payments API)]
  D2[(Search API)]
  D3[(Analytics API)]

  C --> P1 --> D1
  C --> P2 --> D2
  C --> P3 --> D3

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class C client;
  class P1,P2,P3 service;
  class D1,D2,D3 datastore;

Without bulkheads, a single shared pool of, say, 100 threads will all block on the slowest dependency. With per-dependency pools, the slow one fills its own pool, returns "pool exhausted" fast, and the rest of the system stays healthy.

Bulkheads pair naturally with circuit breakers: the bulkhead caps concurrent damage, and the breaker stops trying after a failure rate threshold is crossed. Both are basic resilience primitives used in Hystrix, Resilience4j, Polly, and most service meshes.

Problem statement

A checkout service calls five downstream APIs: payments, inventory, pricing, recommendations, analytics. The analytics API starts taking 30 seconds per request. Within 90 seconds, your entire checkout flow is timing out. Explain why, then design isolation so that one bad dependency cannot take down the whole service.

Why a shared pool fails#

Most HTTP clients and servers default to a single shared worker pool: Tomcat threads, an ExecutorService, a connection pool. When analytics slows down, every thread that calls it parks on the socket. New requests for payments arrive but cannot find a free thread. Latency on every endpoint spikes. This is the textbook cascade.

Little's Law explains the math: concurrency = arrival_rate * latency. If analytics arrival rate is 50 RPS and latency rises from 50 ms to 30 s, required concurrency jumps from 2.5 to 1500 threads. A 100-thread pool is now fully blocked by one slow neighbor.

Bulkhead isolation primitives#

There are three common implementations.

flowchart TB
  subgraph THR[Thread pool bulkhead]
    A1[Caller] --> Q1[[Queue<br/>bounded]] --> W1[Worker pool<br/>N threads] --> D1[Dependency]
  end
  subgraph SEM[Semaphore bulkhead]
    A2[Caller] --> S2((Semaphore<br/>permits = N)) --> D2[Dependency]
  end
  subgraph CONN[Connection pool bulkhead]
    A3[Caller] --> CP3[(Pool<br/>per host)] --> D3[Dependency]
  end

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class A1,A2,A3,W1,S2 service;
  class Q1 queue;
  class CP3 service;
  class D1,D2,D3 datastore;

Thread pool bulkhead. Each dependency gets its own ExecutorService plus a bounded queue. Requests for that dependency are submitted there. If the queue is full, the caller gets a fast RejectedExecutionException. Costs: extra threads, context switches, harder thread-locals.

Semaphore bulkhead. A counter of permits gates the call. Caller acquires a permit, calls the dependency on its own thread, releases. Cheaper than threads. Good when calls are already async / non-blocking. Cannot interrupt a hung call (you still need a timeout).

Connection pool bulkhead. The simplest form: configure HTTP / DB clients with a per-host pool size (maxConnectionsPerRoute in Apache HttpClient, maxIdleConnsPerHost in Go's net/http). Effectively a semaphore on the wire.

Architecture sketch#

flowchart LR
  C[Checkout handler]
  subgraph Bulkheads[Per-dependency bulkheads]
    BP[Payments<br/>pool=20, queue=10]
    BI[Inventory<br/>pool=30, queue=20]
    BR[Recommendations<br/>pool=10, queue=0]
    BA[Analytics<br/>pool=5, queue=0]
  end
  PAY[(Payments)]
  INV[(Inventory)]
  REC[(Recommendations)]
  ANL[(Analytics)]

  C --> BP --> PAY
  C --> BI --> INV
  C -. fire-and-forget .-> BR --> REC
  C -. fire-and-forget .-> BA --> ANL

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class C client;
  class BP,BI,BR,BA service;
  class PAY,INV,REC,ANL datastore;

Critical-path dependencies (payments, inventory) get generous pools. Non-critical ones (recommendations, analytics) get tiny pools and queue=0 so they fail fast and the page can render without them.

Sizing: how big should each pool be?#

Apply Little's Law per dependency:

pool_size = target_rps * p99_latency_seconds

If payments peaks at 200 RPS with p99 = 80 ms, pool size is 200 * 0.08 = 16. Add 25% headroom for variance, settle on 20. Queue size depends on tolerance: a small queue (5 to 10) absorbs micro-bursts, a large queue masks a problem and adds latency.

For non-critical work, undersize on purpose. The analytics pool is 5 not because that is what it needs but because that is the maximum damage you are willing to absorb.

Code example (Resilience4j, Java)#

import io.github.resilience4j.bulkhead.ThreadPoolBulkheadConfig;
import io.github.resilience4j.bulkhead.ThreadPoolBulkhead;
import java.util.concurrent.CompletionStage;

ThreadPoolBulkheadConfig paymentsCfg = ThreadPoolBulkheadConfig.custom()
    .maxThreadPoolSize(20)
    .coreThreadPoolSize(20)
    .queueCapacity(10)
    .build();

ThreadPoolBulkhead paymentsBulkhead =
    ThreadPoolBulkhead.of("payments", paymentsCfg);

CompletionStage<PaymentResponse> result =
    paymentsBulkhead.executeSupplier(() -> paymentsClient.charge(order));

// If pool + queue full, returns a failed future with
// BulkheadFullException - caller can fall back or 503 fast.

Same pattern in Go with a semaphore:

import "golang.org/x/sync/semaphore"

var paymentsSem = semaphore.NewWeighted(20)

func charge(ctx context.Context, o Order) (Resp, error) {
    if err := paymentsSem.Acquire(ctx, 1); err != nil {
        return Resp{}, ErrBulkheadFull
    }
    defer paymentsSem.Release(1)
    return paymentsClient.Charge(ctx, o)
}

Bulkhead vs circuit breaker (complementary, not competing)#

Aspect Bulkhead Circuit Breaker
Protects against Resource exhaustion (slow neighbor) Wasted calls to a known-bad downstream
Trigger Concurrent in-flight > limit Error rate or slow-call rate > threshold
Effect Reject new calls until permit free Reject all calls until timeout elapses
State Stateless counter Closed / open / half-open

You want both. The bulkhead caps the blast radius while the breaker decides to stop trying entirely.

Gotchas#

  • Async leakage. A semaphore bulkhead protects the call site but not the OS thread behind it. Always combine with a request timeout so a hung TCP read does not pin a worker forever.
  • Per-host vs per-route. If your "downstream" is actually a load-balanced cluster of 10 nodes, a single pool of 20 means 2 conns per node. Set per-host pools at the client.
  • Tenant bulkheads. Multi-tenant systems should also bulkhead by tenant ID so one noisy customer cannot exhaust shared capacity. Cell-based architecture is bulkheading taken to the deployment level.
  • Observability is mandatory. Emit bulkhead.queue.depth, bulkhead.permits.used, bulkhead.rejected.count per pool. Without this you cannot tell a starved pool from a healthy one.
  • Do not bulkhead inside a bulkhead. Nested pools usually mean nested timeouts and nested retries, which compounds latency. Pick one isolation layer per hop.

Quick reference#

One-line definition#

A bulkhead is a dedicated, bounded pool of resources (threads, permits, connections) per dependency or tenant. When the pool is exhausted, calls fail fast instead of starving unrelated work.

Three flavors#

  • Thread pool bulkhead: own ExecutorService + bounded queue per dependency. Heavy but isolates blocking IO.
  • Semaphore bulkhead: counter of permits. Cheap, async-friendly, requires explicit timeouts.
  • Connection pool bulkhead: per-host limits on HTTP / DB clients. The default for most prod systems.

Sizing#

  • pool_size = target_rps * p99_latency_sec (Little's Law) plus 20 to 30% headroom.
  • Critical path: size for peak. Non-critical: undersize on purpose.
  • Queue capacity: 0 for "fail now", small (5 to 10) for burst absorption, never unbounded.

Bulkhead vs circuit breaker#

  • Bulkhead caps in-flight calls (resource isolation).
  • Breaker stops calls when downstream is unhealthy (error isolation).
  • Use both. They protect against different failure modes.

Where bulkheads live in practice#

  • Hystrix (archived), Resilience4j, Polly, Failsafe libraries.
  • Envoy / Istio: max_connections, max_pending_requests, max_requests per upstream cluster.
  • Apache HttpClient maxConnectionsPerRoute; Go http.Transport.MaxConnsPerHost.
  • Kubernetes resource requests/limits are pod-level bulkheads against the node.

Anti-patterns#

  • One shared thread pool for every downstream (the cascade default).
  • Unbounded queues "to be safe" - OOM time bomb.
  • Forgetting timeouts on semaphore bulkheads (permits leak on hung sockets).
  • Per-pool tuning without metrics - guessing sizes is worse than the default.
  • Bulkhead chained inside bulkhead - double accounting, hidden latency.

Interview prompts#

  • Q: "Why not just one big pool?" A: One slow dependency starves it; Little's Law shows required concurrency explodes.
  • Q: "How do you choose pool size?" A: From measured p99 latency and peak RPS, plus headroom; verify by load test.
  • Q: "How does this differ from circuit breaker?" A: Bulkhead bounds concurrency; breaker bounds attempts on a sick downstream. Complementary.
  • Q: "What metrics do you emit?" A: permits.used, queue.depth, rejected.count, wait.duration per pool.

Refs#

  • Michael Nygard, Release It! (2nd ed., 2018) - introduces Bulkhead and Circuit Breaker patterns.
  • Resilience4j docs, Bulkhead module (2023).
  • Envoy Proxy docs, Circuit breaking (2024) - bulkhead config at the proxy layer.
  • Marc Brooker (AWS), "Caution: Decreasing load can increase latency" (2017).

FAQ#

What is the bulkhead pattern?#

Bulkhead isolates resources like thread pools, connections, or semaphores per dependency or tenant. If one downstream hangs, it cannot exhaust resources needed by other paths in the service.

Bulkhead vs circuit breaker, what is the difference?#

A bulkhead limits how much capacity a single dependency can consume. A circuit breaker trips fast on errors and stops calls altogether. They complement each other in resilient microservices.

How big should a bulkhead pool be?#

Size it from Little's Law: pool size equals expected throughput times mean response time, plus headroom for tail latency. Too large and isolation is weak, too small and you reject healthy traffic.

When should I use the bulkhead pattern?#

Use it when one service calls multiple downstreams of different criticality, or when serving multiple tenants whose noisy-neighbor traffic could starve others. Critical paths get their own protected pool.

Which libraries implement the bulkhead pattern?#

Resilience4j has a Bulkhead module with thread-pool and semaphore variants. Polly in .NET, Hystrix legacy, and Istio's connection pool limits all implement equivalent isolation.