Skip to content

Background Job Processing#

Problem statement (interviewer prompt)

Design a background-job processing system (Celery / Sidekiq scale). Producers enqueue jobs, workers process them with retries + backoff + DLQ, support delayed jobs, periodic jobs, prioritisation, and 100k+ jobs/s with at-least-once delivery + idempotent handlers.

flowchart LR
  APP[App]
  Q[(Job queue)]
  W([Workers])
  RES[(Result store)]
  APP --> Q --> W --> RES

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class APP service;
    class Q,RES datastore;
    class W compute;

flowchart TB
  subgraph Producers
    APP[App]
    CRON[Periodic]
    API([API trigger])
  end

  subgraph Broker
    Q[(Redis / RabbitMQ / SQS)]
    PRIO[[Priority queues]]
    DELAY([Delayed / scheduled set])
    DLQ[(DLQ)]
  end

  subgraph Workers
    POOL([Worker pool / threadpool])
    AUTO[[Auto-scale by queue depth]]
    HEART[Heartbeats]
    GRACEFUL[Graceful shutdown]
  end

  subgraph Job[Job lifecycle]
    SUB[[Enqueue]]
    DEQ[[Dequeue + ack]]
    EXEC[Execute idempotently]
    RETRY[Retry policy]
    SUCC[Success]
    FAIL[Failure]
  end

  subgraph Ops
    UI([Web UI / Sidekiq Dashboard])
    METR[Metrics]
    ALERT[[Queue depth alert]]
  end

  Producers --> Broker --> Workers
  Workers --> Job
  Job --> RES[(Result store)]
  Job --> DLQ
  Ops --- Broker

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class APP,CRON,HEART,GRACEFUL,EXEC,RETRY,SUCC,FAIL service;
    class DLQ,RES datastore;
    class Q cache;
    class PRIO,AUTO,SUB,DEQ,ALERT queue;
    class API,DELAY,POOL compute;
    class UI,METR obs;

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD Idempotency & retries safe re-execution, backoff + jitter idempotency-retries
HLD Observability metrics, logs, traces, SLOs observability

Quick reference#

Functional#

  • Enqueue jobs from app.
  • Workers consume + execute.
  • Retries, delayed jobs, scheduled jobs.
  • DLQ + replay UI.

Non-functional#

  • Tens of millions of jobs/day common.
  • Per-job latency depends entirely on workload.

Trade-offs#

  • Redis-backed (Sidekiq) = fast, ephemeral; DB-backed = durable.
  • Per-app queues vs shared: isolate noisy neighbors.
  • Autoscaling based on queue depth + processing time.

Refs#

  • Sidekiq / Celery / RQ / Resque docs.
  • "Latency sensitive background jobs" Shopify blog.
  • AWS SQS + Lambda patterns.

FAQ#

How does a background job system work?#

Producers enqueue jobs into a durable queue, workers pull jobs and execute handlers, retries with backoff handle transient failures, and a dead-letter queue captures permanently failing jobs.

What is a dead letter queue?#

A dead letter queue collects jobs that exceeded their retry budget. Operators inspect, fix, and replay them so failed work is not silently lost.

Why use exponential backoff for job retries?#

Exponential backoff with jitter avoids hammering a degraded downstream and prevents synchronized retry storms when many workers fail at the same time.

How does at-least-once delivery affect job handlers?#

At-least-once delivery means a job may run more than once. Handlers must be idempotent, using natural keys or job IDs to deduplicate side effects.

How do background job systems implement priorities?#

Systems use separate queues per priority class, weighted round-robin polling by workers, or a single queue with a sortable score so urgent work is picked first.

How are delayed and periodic jobs implemented?#

Delayed jobs sit in a sorted set keyed by run time and a poller moves due jobs to the active queue. Periodic jobs are enqueued by a leader-elected cron scheduler.