Skip to content

Connection Pool (LLD)#

A connection pool LLD design wraps an expensive resource (TCP socket to a database, RPC channel, HTTPS client) behind borrow() and return(). The pool caps in-flight connections, blocks borrowers when at capacity, validates idle entries before lending, and reaps stale ones. Same pattern as a thread pool, but the resource is heavy: opening a Postgres connection costs ~50ms and ~10MB.

classDiagram
  class ConnectionPool {
    -idle : Deque~Conn~
    -inUse : Set~Conn~
    -semaphore : Semaphore
    -maxSize : int
    -maxIdleMs : long
    -validationQuery : string
    +borrow(timeout) Conn
    +returnConn(Conn) void
    +shutdown() void
  }
  class Conn {
    -socket : Socket
    -createdAt : long
    -lastUsedAt : long
    +validate() bool
    +close() void
  }
  class LeakDetector {
    -borrowTimestamps : Map~Conn, long~
    +onBorrow(Conn)
    +onReturn(Conn)
    +scan() List~Conn~
  }
  ConnectionPool --> Conn : owns N
  ConnectionPool --> LeakDetector

The Semaphore with maxSize permits enforces the cap without polling: borrowers acquire() (block) and release() (wake one). When the pool is exhausted, a borrower can wait with a timeout and throw if no permit arrives in time.

Compared to a thread pool: both bound a scarce resource via a queue, but a connection pool reuses long-lived objects (connections are stateful, must be validated and returned cleanly) while a thread pool reuses workers that pull tasks. Their interfaces differ; their internals are siblings.

classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;

Connection pool LLD design covers everything from borrow/return semantics to leak detection and fairness policies. HikariCP, c3p0, and BoneCP are the production implementations; this guide builds a minimal one with the right primitives and points out where they diverge.

Problem statement (interviewer prompt)

Design a generic connection pool. Support borrow(timeoutMs) -> Connection and release(conn). Cap max connections; block (with timeout) on exhaustion; validate idle entries; reap connections idle longer than maxIdleMs; detect leaks (borrowed but never returned). Discuss FIFO vs LIFO fairness and compare with a thread pool.

Requirements#

Functional

  • borrow(timeoutMs): get a connection, block if none free, throw on timeout.
  • release(conn): return a connection (validates, then re-pools or discards).
  • Configurable maxSize, minIdle, maxIdleMs, validationQuery, leakDetectionMs.
  • shutdown(): drain, close all.

Non-functional

  • O(1) borrow on hot path (no scan).
  • Fair behaviour under contention (configurable).
  • No leak grows the pool indefinitely.

Class design#

classDiagram
  class ConnectionPool {
    -factory : ConnectionFactory
    -idle : Deque~PooledConn~
    -inUse : Set~PooledConn~
    -permits : Semaphore
    -lock : ReentrantLock
    -maxSize : int
    -minIdle : int
    -maxIdleMs : long
    -validationMs : long
    -leakDetectionMs : long
    -reaper : ScheduledExecutorService
    +borrow(timeoutMs) Connection
    +release(Connection)
    +shutdown()
  }
  class PooledConn {
    -delegate : Connection
    -createdAt : long
    -lastUsedAt : long
    -borrowedAt : long
    +validate() bool
    +closePhysical()
  }
  class ConnectionFactory {
    <<interface>>
    +create() Connection
  }
  class LeakDetector {
    -borrowed : Map~PooledConn, long~
    +onBorrow(PooledConn)
    +onRelease(PooledConn)
    +scan(now) List~PooledConn~
  }
  ConnectionPool --> PooledConn : owns
  ConnectionPool --> ConnectionFactory
  ConnectionPool --> LeakDetector
  PooledConn --> Connection : wraps

PooledConn is a wrapper, not the raw Connection. It carries timestamps and a back-pointer to the pool so close() on the wrapper returns rather than closes.

Core flow#

sequenceDiagram
  participant C as Caller
  participant P as Pool
  participant S as Semaphore
  participant Q as IdleDeque
  C->>S: acquire(timeout)
  alt no permit in time
    S-->>C: timeout exception
  else permit granted
    S-->>C: ok
    C->>Q: pollFirst (LIFO) or pollLast (FIFO)
    alt idle exists
      Q-->>C: PooledConn
      C->>C: validate()
      alt invalid
        C->>C: close physical, create new
      end
    else empty
      C->>C: factory.create
    end
    C-->>C: use connection
    C->>P: release(conn)
    P->>Q: push back
    P->>S: release()
  end

The permit acquisition gates capacity; the deque holds idle entries; the fast path is permit + poll + (maybe) validate.

Code#

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;

public class ConnectionPool implements AutoCloseable {
    private final ConnectionFactory factory;
    private final Deque<PooledConn> idle = new ArrayDeque<>();
    private final Set<PooledConn> inUse = Collections.newSetFromMap(new ConcurrentHashMap<>());
    private final Semaphore permits;
    private final ReentrantLock lock = new ReentrantLock(false);  // unfair = LIFO-ish, faster
    private final int maxSize;
    private final long maxIdleMs;
    private final ScheduledExecutorService reaper =
        Executors.newSingleThreadScheduledExecutor();
    private volatile boolean closed = false;

    public ConnectionPool(ConnectionFactory f, int maxSize, long maxIdleMs) {
        this.factory = f;
        this.maxSize = maxSize;
        this.maxIdleMs = maxIdleMs;
        this.permits = new Semaphore(maxSize, false);
        reaper.scheduleWithFixedDelay(this::reapIdle, 30, 30, TimeUnit.SECONDS);
    }

    public Connection borrow(long timeoutMs) throws InterruptedException {
        if (closed) throw new IllegalStateException("pool closed");
        if (!permits.tryAcquire(timeoutMs, TimeUnit.MILLISECONDS)) {
            throw new TimeoutException("no connection available in " + timeoutMs + "ms");
        }
        PooledConn pc;
        lock.lock();
        try { pc = idle.pollFirst(); }  // LIFO: hot cache, recently used
        finally { lock.unlock(); }
        if (pc == null) pc = new PooledConn(factory.create());
        if (!pc.validate()) {
            pc.closePhysical();
            pc = new PooledConn(factory.create());
        }
        pc.borrowedAt = System.nanoTime();
        inUse.add(pc);
        return pc;
    }

    public void release(PooledConn pc) {
        if (!inUse.remove(pc)) return;  // double-release guard
        pc.lastUsedAt = System.nanoTime();
        pc.borrowedAt = 0;
        if (closed || !pc.validate()) {
            pc.closePhysical();
        } else {
            lock.lock();
            try { idle.offerFirst(pc); }
            finally { lock.unlock(); }
        }
        permits.release();
    }

    private void reapIdle() {
        long cutoff = System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(maxIdleMs);
        List<PooledConn> dead = new ArrayList<>();
        lock.lock();
        try {
            Iterator<PooledConn> it = idle.descendingIterator();
            while (it.hasNext()) {
                PooledConn p = it.next();
                if (p.lastUsedAt < cutoff) { it.remove(); dead.add(p); }
            }
        } finally { lock.unlock(); }
        for (PooledConn p : dead) p.closePhysical();
    }

    @Override
    public void close() {
        closed = true;
        reaper.shutdown();
        lock.lock();
        try {
            for (PooledConn p : idle) p.closePhysical();
            idle.clear();
        } finally { lock.unlock(); }
    }
}

The lock guards the deque only; the semaphore guards the count. Two locks for two invariants is cleaner than one big mutex.

FIFO vs LIFO fairness#

Policy Behaviour Pro Con
LIFO (default) pollFirst + offerFirst Hot connections stay hot; idle ones age out and get reaped Some connections never used until pool shrinks
FIFO pollFirst + offerLast Round-robin across all connections None ever get reaped if traffic is steady
Fair semaphore new Semaphore(n, true) First waiter wins Lock contention, slower

HikariCP picks LIFO with a custom unfair queue, exploiting JVM thread parking to keep the hot connection hot. For most workloads LIFO is the right default.

Validation and leak detection#

Validation: before lending, run SELECT 1 or check socket liveness. Expensive if synchronous; HikariCP uses TCP isValid(timeout) (driver-level, sub-ms).

Leak detection: when borrowing, stamp borrowedAt. The reaper scans every 30s; any inUse connection held longer than leakDetectionMs (say 60s) logs a stack trace and (optionally) force-recovers.

class LeakDetector {
    private final Map<PooledConn, StackTraceElement[]> borrowed = new ConcurrentHashMap<>();

    void onBorrow(PooledConn p) {
        borrowed.put(p, Thread.currentThread().getStackTrace());
    }

    void onRelease(PooledConn p) {
        borrowed.remove(p);
    }

    void scan(long thresholdNanos) {
        long now = System.nanoTime();
        borrowed.forEach((p, stack) -> {
            if (now - p.borrowedAt > thresholdNanos) {
                log.warn("LEAK suspected, borrowed at: {}", Arrays.toString(stack));
            }
        });
    }
}

The stack trace lets you find the calling code that forgot to release(). Production code wraps borrow in try-with-resources to make leaks impossible.

Concurrency considerations#

  • Semaphore is the bottleneck enforcer: do not protect it with a lock.
  • ConcurrentHashMap.newKeySet() for inUse avoids locking on every borrow/release.
  • Lock granularity: the deque is the only thing under lock. Validation and physical close happen outside the lock (slow, do not block borrowers).
  • Reaper thread: a single scheduled executor, no per-pool thread explosion.
  • Double-release: inUse.remove(pc) returns false if already released; guard prevents permit double-release.

Comparison with thread pool#

Aspect Thread pool Connection pool
Bounded resource Threads Sockets / connections
Pulled by Workers from a queue Caller via borrow
Returned by Worker loop after task Caller via release
Validation N/A Required (socket can die)
Idle timeout keepAliveTime evicts maxIdleMs reaps
Failure mode OOM, thread starvation Leak grows pool, then exhaustion
Saturation policy Rejection handler Block with timeout, then throw

Both use a BlockingQueue-or-Semaphore cap and a worker pattern. Connection pools add validation and leak detection because the resource is fragile and external.

Edge cases interviewers probe#

  • A connection's underlying TCP socket closes while idle: validation must detect (some drivers raise on next query, so use TCP keepalive or isValid).
  • Pool min size: pre-create minIdle connections on startup to avoid first-request latency.
  • Async borrow: return a CompletableFuture<Connection>; the permit acquire happens off the caller's thread.
  • Multi-tenant: per-tenant sub-pools, or a single pool with priority queue.
  • release() from a different thread than borrow(): must be safe; that is the common case (work passed between threads).

Where connection pools fit#

flowchart LR
  CP_LLD((Connection<br/>pool LLD))
  CP_HLD[Connection pooling<br/>HLD framing]
  OP[Object pool pattern<br/>generalization]
  TP[Thread pool executor<br/>cousin]
  CP_HLD -. concretizes .-> CP_LLD
  OP -. parent .- CP_LLD
  TP -. cousin .- CP_LLD

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class CP_LLD service;
    class CP_HLD,OP,TP datastore;

Quick reference#

Key classes#

  • ConnectionPool: orchestrator
  • PooledConn: wraps raw Connection, adds timestamps + back-pointer
  • ConnectionFactory: creates raw connections (DB driver, gRPC channel)
  • LeakDetector: tracks borrow stack traces
  • ScheduledExecutorService: reaper thread

Borrow flow#

  1. semaphore.tryAcquire(timeoutMs); throw on timeout
  2. idle.pollFirst() (LIFO) or create new via factory
  3. validate(); if invalid, close + create fresh
  4. Stamp borrowedAt, add to inUse, return

Release flow#

  1. Remove from inUse (double-release guard)
  2. Validate again; if invalid, close physical
  3. idle.offerFirst(pc)
  4. semaphore.release()

Reaper#

  • Scheduled every 30s
  • Scans tail of idle deque (oldest); closes if lastUsedAt < now - maxIdleMs
  • Keeps minIdle minimum

Fairness#

Policy Pull from Push to Notes
LIFO front front hot stays hot; default in HikariCP
FIFO front back round-robin; nothing ever idle-reaped
Fair semaphore n/a n/a first-waiter wins; slow under contention

Validation strategies#

  • SELECT 1: portable but expensive (ms)
  • Connection.isValid(timeout): driver-native, sub-ms
  • TCP keepalive: passive, catches dead sockets between borrows
  • On-checkout (default), on-checkin, on-idle (different policies)

Leak detection#

  • Stamp borrowedAt on every borrow
  • Reaper scans inUse; flag entries held > threshold
  • Capture stack trace at borrow to find the culprit
  • Production: try-with-resources to make leaks impossible

Complexity#

Op Time
borrow (fast path) O(1)
borrow (cold start) O(create + validate)
release O(1)
reapIdle O(idle) every 30s
shutdown O(all)

Concurrency tools#

  • Semaphore(maxSize): hard cap on borrowers
  • ReentrantLock: protects the idle deque only
  • ConcurrentHashMap.newKeySet(): lock-free inUse tracking
  • ScheduledExecutorService: reaper

Thread pool vs connection pool#

Dimension Thread pool Connection pool
Resource thread socket
Pulled by worker caller
Validation no yes
Failure OOM leak
Saturation reject block-then-timeout

Interview questions to expect#

  • Why not just open a new connection per request? ~50ms + 10MB; doesn't scale
  • LIFO vs FIFO? LIFO keeps cache hot, lets idle ones age out and get reaped
  • How do you detect leaks? stack trace at borrow + scan inUse periodically
  • What happens if release is missed? leak; semaphore permit never returned
  • Async borrow? wrap in CompletableFuture; permit acquire on the event loop

Refs#

  • HikariCP wiki - design choices
  • c3p0 connection pool docs
  • LMAX disruptor (lock-free queue inspiration)
  • AWS RDS Proxy (managed pool service)

FAQ#

How does a connection pool cap concurrent connections?#

It uses a counting semaphore initialised with maxSize permits, so borrowers block on acquire when the pool is exhausted and release returns a permit.

Why is LIFO usually preferred over FIFO for connection pools?#

LIFO keeps the most recently used connection hot in cache and lets older idle entries age out and get reaped, which improves locality and helps the reaper shrink the pool.

How are connection leaks detected?#

Stamp a borrowedAt timestamp and capture the borrow stack trace; a reaper periodically scans inUse and logs any connection held longer than the leak threshold.

How is connection validation done cheaply?#

Use the driver-native Connection.isValid(timeout) call, which is typically sub-millisecond, instead of running SELECT 1 on every borrow.

How does a connection pool differ from a thread pool?#

Both bound a scarce resource but connection pools wrap stateful sockets that need validation and leak detection, while thread pools run stateless workers that pull tasks from a queue.

What happens when release is never called?#

The connection stays in inUse and its semaphore permit never returns, so the pool eventually exhausts and new borrowers time out. The leak detector exists to surface this fast.