Job / Task Scheduler#

Problem statement (interviewer prompt)

Design a distributed cron / job scheduler (Airflow / Quartz-cluster / a giant cron). Run tens of thousands of scheduled jobs across many workers with exactly-once-per-schedule semantics, retries, DAG dependencies, and resumability on worker failure.

flowchart LR
  USR([User / Cron entry])
  SCH([Scheduler])
  Q[(Job Queue)]
  W([Workers])
  DB[(Job state DB)]
  USR --> SCH --> Q --> W
  W --> DB

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Q,DB datastore;
    class USR,SCH,W compute;

flowchart TB
  subgraph Author
    DAG[DAG / workflow definition]
    GIT[Git repo]
    UI([Web UI])
  end

  subgraph Sched[Scheduler]
    PARSE([DAG parser])
    PLAN([Run planner<br/>schedules + backfill])
    LEAD[Leader election - HA]
    LOCK[Distributed lock per DAG run]
    TIME([Cron evaluator])
  end

  subgraph Queue
    Q[[Queue per worker pool]]
    PRIO[[Priority queues]]
    DLQ[Dead-letter]
  end

  subgraph Workers
    W1([Worker / executor])
    W2([Worker])
    KEXEC[K8s executor]
    CEXEC[Celery executor]
    SUB[Sub-process / pod per task]
  end

  subgraph State[State + History]
    DB[(Metadata DB)]
    LOG[Task logs]
    TS([Trigger / event store])
    ART[Artifacts]
  end

  subgraph Reliability
    RETRY[Retry policies + backoff]
    SLA[SLA miss alerts]
    IDEMP[Idempotency for tasks]
    CKPT[Checkpoint long tasks]
  end

  subgraph Trigger
    CRON([Cron schedule])
    SENS([Sensors / external triggers])
    API([On-demand trigger API])
  end

  Author --> Sched
  Sched --> Queue --> Workers
  Workers --> State
  Reliability --- Sched
  Trigger --- Sched

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class UI client;
    class DAG,GIT,LEAD,LOCK,DLQ,KEXEC,CEXEC,SUB,ART,RETRY,IDEMP,CKPT service;
    class DB datastore;
    class Q,PRIO queue;
    class PARSE,PLAN,TIME,W1,W2,TS,CRON,SENS,API compute;
    class LOG,SLA obs;

Correctness patterns#

Singleton scheduler: leader election in HA pair to avoid double-runs.
Idempotent tasks: each run keyed by (dag, run_id, task_id, attempt).
Workers ack work: re-queue on heartbeat loss; tasks must tolerate at-least-once.
Backfill = scheduling historical runs after deploy.

Workflow systems vs cron#

Simple cron: timer + job command.
Workflow systems (Airflow / Argo / Temporal / Cadence) add DAGs, retries, sensors, observability, durable state.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
`HLD`	Raft / Paxos consensus	replicated state machine via majority quorum	consensus-raft-paxos
`HLD`	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`HLD`	Event sourcing + CQRS	commands -> events; separate read model	event-sourcing-cqrs
`LLD`	Creational patterns	Singleton, Factory, Builder, Prototype	creational-patterns
`LLD`	Behavioural patterns	Strategy, Observer, State, Command, Chain	behavioral-patterns

Quick reference#

Functional#

Schedule jobs by cron or event.
Dependencies / DAGs.
Retries, SLAs, alerts.
Backfill / rerun.
Operator UI + API.

Non-functional#

Sub-minute scheduling precision.
HA scheduler with leader election.
Tens of thousands of concurrent task runs.

Trade-offs#

Airflow = batch + DAG-first; Temporal/Cadence = code-first long workflows; Argo = K8s-native.
Pull workers vs push from scheduler: pull preferred for scale.
At-least-once execution = task author MUST design idempotency.

Refs#

Apache Airflow docs; Temporal, Cadence; Argo Workflows.
Google "Borg" paper (cluster scheduling cousin).
Netflix Conductor docs.

FAQ#

How does a distributed job scheduler work?#

A leader-elected scheduler reads cron specs, computes the next run time, and pushes ready jobs onto a queue. Workers claim jobs, run them, and report status back.

How does the scheduler achieve exactly-once-per-schedule semantics?#

Schedulers persist a unique fire-time identifier per job and use a conditional write so concurrent schedulers cannot enqueue the same fire-time twice.

What is the difference between a job scheduler and an orchestrator?#

A scheduler triggers jobs on time-based or event-based rules. An orchestrator manages multi-step workflows with task-level dependencies, retries, and data passing.

How does a scheduler handle worker failures?#

Workers heartbeat to the scheduler. If a worker dies mid-job, the scheduler reassigns the job to another worker after the lease expires, and idempotent handlers prevent duplicates.

How are DAG dependencies enforced?#

The scheduler tracks parent state for each task. A child becomes runnable only when all its parents succeed; failed parents either skip the child or trigger a fail-fast policy.

Why use leader election for the scheduler?#

A single leader avoids duplicate enqueues. Leader election via ZooKeeper or etcd ensures exactly one scheduler is active while standbys are ready to take over.