Skip to content

Thread Pool Executor#

A thread pool executor LLD is the bridge between submitting tasks and bounded concurrent execution. A pool of worker threads sleep on a shared blocking queue; submitted Runnable / Callable tasks land on the queue; workers wake, drain, run, and loop. The size cap prevents thread explosion; the queue cap prevents memory explosion.

classDiagram
  class Executor {
    <<interface>>
    +execute(task) void
  }
  class ThreadPoolExecutor {
    -corePoolSize : int
    -maxPoolSize : int
    -workQueue : BlockingQueue~Runnable~
    -workers : Set~Worker~
    -rejectionHandler : RejectedExecutionHandler
    +execute(Runnable) void
    +submit(Callable) Future
    +shutdown() void
    +shutdownNow() List~Runnable~
  }
  class Worker {
    -thread : Thread
    -task : Runnable
    +run() void
  }
  class Future~T~ {
    +get() T
    +cancel(mayInterrupt) bool
    +isDone() bool
  }
  Executor <|-- ThreadPoolExecutor
  ThreadPoolExecutor --> Worker : owns N
  ThreadPoolExecutor --> Future : returns

Three classic variants ship with every JDK and Python concurrent.futures:

  • Fixed pool: N workers, unbounded queue. Memory grows under burst.
  • Cached pool: 0 to Integer.MAX_VALUE workers, SynchronousQueue (no buffering). Threads explode under burst.
  • Scheduled pool: priority queue keyed by run-at time; workers poll the head.

When the queue is full and the pool is saturated, the rejection policy kicks in: AbortPolicy throws, CallerRunsPolicy runs on the submitting thread (back-pressure), DiscardPolicy silently drops, DiscardOldestPolicy evicts the head.

classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;

The thread pool executor LLD is one of the most-asked concurrency interviews. It exercises bounded queues, worker threads, rejection policies, Future semantics, and graceful shutdown: a checklist of every concurrency primitive worth knowing. This guide builds the design from scratch and ends with work-stealing variants.

Problem statement (interviewer prompt)

Design a thread pool executor. Support execute(Runnable), submit(Callable) -> Future, configurable core/max pool size, a bounded work queue, pluggable rejection policy, and a graceful shutdown() plus immediate shutdownNow(). Discuss fixed vs cached vs scheduled variants and where work-stealing helps.

Requirements#

Functional

  • execute(Runnable): fire-and-forget.
  • submit(Callable<T>) -> Future<T>: result-bearing.
  • Configurable corePoolSize, maxPoolSize, keepAliveTime, workQueue, rejectionHandler.
  • shutdown(): stop accepting new tasks, drain queue.
  • shutdownNow(): interrupt workers, return undrained tasks.
  • awaitTermination(timeout).

Non-functional

  • Bounded memory (cap the queue).
  • Predictable back-pressure under burst.
  • No goroutine/thread explosion.

Class design#

classDiagram
  class Executor {
    <<interface>>
    +execute(Runnable)
  }
  class ExecutorService {
    <<interface>>
    +submit(Callable) Future
    +shutdown()
    +shutdownNow() List
    +awaitTermination(timeout) bool
  }
  class AbstractExecutorService
  class ThreadPoolExecutor {
    -corePoolSize : int
    -maxPoolSize : int
    -keepAliveNanos : long
    -workQueue : BlockingQueue~Runnable~
    -workers : HashSet~Worker~
    -mainLock : ReentrantLock
    -termination : Condition
    -ctl : AtomicInteger
    -handler : RejectedExecutionHandler
    +execute(Runnable)
  }
  class Worker {
    -thread : Thread
    -firstTask : Runnable
    -completedTasks : long
    +run()
    +runWorker()
  }
  class FutureTask~T~ {
    -state : int
    -result : Object
    +run()
    +get() T
    +cancel(bool) bool
  }
  class RejectedExecutionHandler {
    <<interface>>
    +rejectedExecution(Runnable, ThreadPoolExecutor)
  }
  Executor <|-- ExecutorService
  ExecutorService <|.. AbstractExecutorService
  AbstractExecutorService <|-- ThreadPoolExecutor
  ThreadPoolExecutor --> Worker
  ThreadPoolExecutor --> FutureTask : wraps Callable
  ThreadPoolExecutor --> RejectedExecutionHandler

The interface hierarchy mirrors java.util.concurrent. AbstractExecutorService provides the default submit that wraps a Callable in a FutureTask and delegates to execute.

Core flow#

sequenceDiagram
  participant C as Caller
  participant E as ThreadPoolExecutor
  participant Q as BlockingQueue
  participant W as Worker
  C->>E: submit(callable)
  E->>E: wrap as FutureTask
  alt workers < core
    E->>W: addWorker(task)
    W->>W: runWorker (run task immediately)
  else queue not full
    E->>Q: offer(task)
  else workers < max
    E->>W: addWorker(task)
  else
    E->>E: reject(task)
  end
  loop worker loop
    W->>Q: take() (blocks if empty)
    Q-->>W: task
    W->>W: task.run()
    W->>W: completedTasks++
  end

The four-branch decision tree on submission is the heart of the implementation. The order matters: try core threads first, then queue, then extra threads up to max, then reject.

Code#

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.concurrent.locks.*;

public class MyThreadPoolExecutor extends AbstractExecutorService {
    private final int corePoolSize, maxPoolSize;
    private final long keepAliveNanos;
    private final BlockingQueue<Runnable> workQueue;
    private final HashSet<Worker> workers = new HashSet<>();
    private final ReentrantLock mainLock = new ReentrantLock();
    private final Condition termination = mainLock.newCondition();
    private final AtomicInteger activeCount = new AtomicInteger(0);
    private volatile boolean shutdown = false;
    private final RejectedExecutionHandler handler;

    public MyThreadPoolExecutor(int core, int max, long keepAliveMs,
                                BlockingQueue<Runnable> q,
                                RejectedExecutionHandler h) {
        this.corePoolSize = core;
        this.maxPoolSize = max;
        this.keepAliveNanos = TimeUnit.MILLISECONDS.toNanos(keepAliveMs);
        this.workQueue = q;
        this.handler = h;
    }

    @Override
    public void execute(Runnable command) {
        if (shutdown) { handler.rejectedExecution(command, null); return; }
        int active = activeCount.get();
        if (active < corePoolSize && addWorker(command, true)) return;
        if (workQueue.offer(command)) return;
        if (!addWorker(command, false)) handler.rejectedExecution(command, null);
    }

    private boolean addWorker(Runnable firstTask, boolean core) {
        int cap = core ? corePoolSize : maxPoolSize;
        if (activeCount.get() >= cap) return false;
        activeCount.incrementAndGet();
        mainLock.lock();
        try {
            Worker w = new Worker(firstTask);
            workers.add(w);
            w.thread.start();
            return true;
        } finally { mainLock.unlock(); }
    }

    private final class Worker implements Runnable {
        final Thread thread;
        Runnable firstTask;
        Worker(Runnable t) { this.firstTask = t; this.thread = new Thread(this); }

        public void run() {
            Runnable task = firstTask; firstTask = null;
            try {
                while (task != null || (task = getTask()) != null) {
                    try { task.run(); } finally { task = null; }
                }
            } finally {
                mainLock.lock();
                try { workers.remove(this); activeCount.decrementAndGet();
                      if (workers.isEmpty()) termination.signalAll(); }
                finally { mainLock.unlock(); }
            }
        }

        Runnable getTask() {
            try {
                if (shutdown && workQueue.isEmpty()) return null;
                return activeCount.get() > corePoolSize
                    ? workQueue.poll(keepAliveNanos, TimeUnit.NANOSECONDS)
                    : workQueue.take();
            } catch (InterruptedException e) { return null; }
        }
    }

    public void shutdown() { shutdown = true; }
    public List<Runnable> shutdownNow() {
        shutdown = true;
        List<Runnable> drained = new ArrayList<>();
        workQueue.drainTo(drained);
        mainLock.lock();
        try { for (Worker w : workers) w.thread.interrupt(); }
        finally { mainLock.unlock(); }
        return drained;
    }
    public boolean isShutdown() { return shutdown; }
    public boolean isTerminated() { return shutdown && workers.isEmpty(); }
    public boolean awaitTermination(long t, TimeUnit u) throws InterruptedException {
        long nanos = u.toNanos(t);
        mainLock.lock();
        try {
            while (!isTerminated()) {
                if (nanos <= 0) return false;
                nanos = termination.awaitNanos(nanos);
            }
            return true;
        } finally { mainLock.unlock(); }
    }
}

The 130 lines above are 95% of what java.util.concurrent.ThreadPoolExecutor does, minus the ctl packed-int bookkeeping and the saturation-policy callbacks. Read the JDK source for the production version.

Rejection policies#

Policy Behaviour When to use
AbortPolicy (default) Throws RejectedExecutionException Caller can react
CallerRunsPolicy Caller thread runs the task inline Natural back-pressure (slows producer)
DiscardPolicy Silently drop Telemetry, best-effort jobs
DiscardOldestPolicy Evict queue head, retry submit Newest-data-wins

CallerRunsPolicy is the most underrated choice: it gives you free back-pressure without explicit signalling.

Pool variants#

flowchart TB
  Fixed[Fixed pool<br/>core = max = N<br/>unbounded LinkedBlockingQueue]
  Cached[Cached pool<br/>core = 0, max = MAX<br/>SynchronousQueue]
  Scheduled[Scheduled pool<br/>DelayedWorkQueue<br/>priority by runAt]
  WS[Work-stealing pool<br/>ForkJoinPool<br/>per-worker deques]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    class Fixed,Cached,Scheduled,WS service;

Fixed pool is the safe default: predictable thread count, but unbounded queue means an OOM risk under burst. Pair with a bounded queue + CallerRunsPolicy for production.

Cached pool is a footgun: a burst of N requests spawns N threads in O(burst) time. Use only for short-lived, well-bounded workloads.

Scheduled pool uses a DelayedWorkQueue (binary heap) keyed by runAt. Workers take() the head and sleep until its time; periodic tasks re-add themselves.

Work-stealing pool (ForkJoinPool) gives each worker its own deque. Workers push/pop from their own deque (no contention); when empty, they steal from another worker's tail. Crucial for divide-and-conquer where task counts explode.

Concurrency considerations#

  • activeCount as AtomicInteger lets the fast path skip the main lock. Only addWorker and shutdown take the lock.
  • BlockingQueue.take() uses ReentrantLock + condition under the hood. Workers block here when idle.
  • Interrupt protocol: shutdownNow() calls Thread.interrupt() on every worker. Workers exit when take() throws InterruptedException.
  • Memory visibility: shutdown is volatile. Workers re-check after every loop iteration.
  • Avoid double-counting: increment activeCount before start(), decrement only when the worker run loop exits.

Edge cases interviewers probe#

  • A task throws an unchecked exception: catch it in the worker loop; do not let it kill the thread.
  • shutdownNow() while a task is running: the task sees Thread.interrupted() but must opt in to abort.
  • Submitting from inside a task (recursion): risk of deadlock if the queue is full and the caller is the only worker.
  • Idle worker reaping: workers above corePoolSize should poll(keepAliveTime) instead of take() so they self-terminate.
  • Pre-starting all core threads: prestartAllCoreThreads() reduces first-request latency.

Where thread pools fit#

flowchart LR
  TP((Thread pool<br/>executor))
  CP[Concurrency primitives<br/>BlockingQueue + Lock]
  AS[Async models<br/>Executor is the runtime]
  TD[Threading and deadlocks<br/>contention failure modes]
  PC[Producer consumer<br/>queue + workers]
  CP --> TP
  AS -. backs .- TP
  TD -. failure modes .-> TP
  TP -. is a .- PC

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class TP service;
    class CP,AS,TD,PC datastore;

Quick reference#

Key classes#

  • Executor / ExecutorService: interfaces
  • AbstractExecutorService: wraps Callable in FutureTask
  • ThreadPoolExecutor: core/max/queue/handler
  • Worker: a Runnable owning a Thread, loops on getTask()
  • FutureTask<T>: state machine for result, cancel, exception
  • RejectedExecutionHandler: pluggable saturation policy

Submission decision tree#

  1. activeCount < corePoolSize then spawn a core worker
  2. else workQueue.offer(task) then enqueue
  3. else activeCount < maxPoolSize then spawn a burst worker
  4. else handler.rejectedExecution(task, this) (reject)

Rejection policies#

Policy What When
AbortPolicy throws RejectedExecutionException default, caller reacts
CallerRunsPolicy inline on submitter natural back-pressure
DiscardPolicy silently drops best-effort
DiscardOldestPolicy evicts head, retries submit newest-wins

Pool variants#

Variant Core Max Queue Watch out for
Fixed N N unbounded LinkedBlockingQueue OOM under burst
Cached 0 MAX SynchronousQueue thread explosion
Scheduled N MAX DelayedWorkQueue (heap) clock-skew
Work-stealing parallelism same per-worker deques only for CPU-bound

Complexity#

Op Time
execute (fast path) O(1)
submit(Callable) O(1) wrap + execute
Future.get() blocks
shutdown O(1)
shutdownNow O(workers) + drain

Concurrency tools used#

  • BlockingQueue<Runnable>: workers wait on take()
  • ReentrantLock (mainLock): protects worker set
  • Condition (termination): awaitTermination waits here
  • AtomicInteger (ctl or activeCount): fast-path worker count
  • volatile boolean shutdown: visibility for state flag

Interview questions to expect#

  • Why core vs max? core stays alive; burst threads above core die after keepAlive
  • Why try queue before max? prefer queueing to spawning extra threads
  • How does shutdown vs shutdownNow differ? shutdown drains; shutdownNow interrupts
  • Why use CallerRunsPolicy? gives back-pressure for free
  • What if a task throws? catch in worker loop or thread dies
  • How does work-stealing differ? per-worker deques + steal-from-tail

Pitfalls#

  • Unbounded queue + fixed pool = silent OOM
  • Executors.newCachedThreadPool in user-facing code = thread explosion
  • Submitting from inside a task with full queue = deadlock
  • Forgetting to awaitTermination = JVM exits while tasks queued

Refs#

  • JDK ThreadPoolExecutor source (openjdk)
  • Goetz - Java Concurrency in Practice ch.8
  • Doug Lea - util.concurrent design (PDF, original 1999 paper)
  • LMAX Disruptor (lock-free variant for ultra-low latency)

FAQ#

How does a thread pool executor work internally?#

Worker threads loop on a shared BlockingQueue; submit pushes a Runnable onto the queue and an idle worker pops and runs it. The pool size caps concurrent execution.

What is the difference between corePoolSize and maxPoolSize?#

core is the steady-state count of workers kept alive. The pool grows up to max when the queue is full, and excess threads above core idle out after keepAliveTime.

What rejection policies exist?#

AbortPolicy (throw), CallerRunsPolicy (run in submitter thread), DiscardPolicy (drop silently), DiscardOldestPolicy (drop head of queue). Pick based on backpressure semantics.

How do you size a thread pool?#

For CPU-bound work use N or N+1 where N is core count. For I/O-bound work use N times (1 + wait/compute ratio). Always measure under realistic load.

When do you choose a cached vs fixed thread pool?#

Fixed bounds the worker count and is safer for predictable load. Cached creates threads on demand and is fine for short, bursty tasks but can explode under sustained load.