Skip to content

Data Pipeline Orchestrator#

Problem statement (interviewer prompt)

Design a workflow orchestrator like Airflow / Dagster / Prefect: developers write DAGs of tasks; the orchestrator schedules them, runs them on a pool of workers, retries failures, supports backfill, and exposes status.

flowchart TB
  Dev[Developer] -->|commit DAG| Repo[(DAG repo)]
  Repo --> Parser[DAG parser]
  Parser --> MetaDB[(Metadata DB:<br/>dags, tasks, runs)]
  Sched[Scheduler] --> MetaDB
  Sched --> EvQ[(Task queue)]
  EvQ --> Exec1[Executor worker]
  EvQ --> Exec2[Executor worker]
  Exec1 --> MetaDB
  Exec1 --> Log[(Log + metric store)]
  UI[Web UI] --> MetaDB
  UI --> Log

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Dev client;
    class UI edge;
    class Parser,Sched service;
    class Exec1,Exec2 compute;
    class MetaDB,Log,Repo datastore;
    class EvQ queue;

The core abstraction is a DAG (directed acyclic graph) of tasks; the scheduler picks tasks whose upstreams have succeeded and pushes them to executor workers.

The orchestrator runs the DAGs that power dashboards, ML training, data lakes, and reverse-ETL. Reliability and visibility win over raw performance.

DAG and task model#

@dag(schedule="@daily")
def daily_revenue():
    raw = ingest_orders()          # task A
    cleaned = clean(raw)           # task B, depends on A
    daily = aggregate(cleaned)     # task C, depends on B
    publish(daily)                 # task D, depends on C

The DAG is parsed once per worker; the metadata DB stores the structure. Each scheduled run materialises tasks with state PENDING.

Scheduler#

flowchart TB
  Tick[Tick every Ns] --> SelectDag[Select DAGs due to run]
  SelectDag --> CreateRun[Create DagRun row]
  CreateRun --> Loop[For each task whose upstream succeeded]
  Loop --> Enqueue[Push to executor queue]
  Loop --> Done[Mark dependencies satisfied]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Tick,SelectDag,CreateRun,Loop,Enqueue,Done service;

Executors#

Type Use
Local Single-host; dev
Celery Workers pull from Redis/RabbitMQ
Kubernetes Each task runs in its own pod
Ray / Dask Distributed compute alongside
Cloud serverless (Step Functions) Fully managed

Retries and SLA#

@task(retries=3, retry_delay=timedelta(minutes=5), execution_timeout=timedelta(hours=1))
def ingest_orders(): ...

Failure → wait → retry. After N failures, mark FAILED; downstream skipped. SLA misses trigger alerts.

Backfill#

"Re-run last 30 days because the transform had a bug."

The scheduler creates DagRuns for past dates and executes tasks for each. Idempotency is the developer's responsibility (MERGE INTO instead of INSERT).

Sensors and data-aware scheduling#

@task
def wait_for_file():
    while not s3.exists(key):
        sleep(60)

Long-running "sensor" tasks block downstream until a precondition holds (file arrives, partition created, API responds).

Modern orchestrators (Dagster, Airflow 2.4+) support data-aware scheduling based on asset / dataset updates instead of cron.

State per task instance#

PENDING → SCHEDULED → QUEUED → RUNNING → SUCCESS | FAILED | UP_FOR_RETRY | SKIPPED

Materialised in task_instance table; UI is essentially a view over this.

Multi-tenant isolation#

  • Per-team / per-project DAG folders.
  • RBAC on UI.
  • Per-team executor pool to bound resource consumption.
  • Per-task resource requests (Kubernetes executor).

Tools#

Tool Notes
Airflow The original; cron-based + sensors
Dagster Software-defined assets, data-aware
Prefect Python-native, dynamic workflows
Argo Workflows K8s-native YAML
Temporal Workflow-as-code, durable execution
AWS Step Functions Fully managed

Where orchestrators fit#

flowchart TB
  DAG((Data pipeline<br/>orchestrator))
  STR[Batch / stream processing<br/>workloads orchestrated]
  LAKE[Data lake / warehouse<br/>typical sink]
  JS[Job scheduler<br/>simpler primitive]
  CO[Container orchestration<br/>K8s executor]
  DAG --> STR
  DAG --> LAKE
  JS -. simpler .- DAG
  CO --> DAG

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class DAG service;
    class STR,LAKE,JS,CO datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Batch stream processing the workloads run batch-stream-processing
HLD Data Lake Warehouse the typical sink data-lake-warehouse
HLD Job Scheduler the simpler sibling job-scheduler
HLD Container Orchestration the K8s executor runtime container-orchestration

Quick reference#

Core abstractions#

  • DAG (directed acyclic graph of tasks)
  • Task (unit of work)
  • DagRun (scheduled instance of a DAG)
  • TaskInstance (state per task per run)

Scheduler tick#

Find DAGs due → create DagRun → for each task with all upstream SUCCESS → enqueue.

Executors#

Type Use
Local Dev
Celery Standard distributed
Kubernetes Per-task pod isolation
Ray / Dask Distributed compute
Step Functions Managed serverless

State#

PENDING → SCHEDULED → QUEUED → RUNNING → SUCCESS | FAILED | RETRY | SKIPPED.

Reliability#

  • Retries with backoff
  • Execution timeout
  • SLA misses trigger alerts
  • Idempotent tasks for safe re-run

Backfill#

Re-create DagRuns for past dates; tasks must be idempotent.

Sensors / data-aware#

Wait on file/partition/asset before running downstream.

Multi-tenant#

Per-team DAG folders, RBAC, executor pools, per-task resource requests.

Tools#

Airflow, Dagster, Prefect, Argo Workflows, Temporal, AWS Step Functions.

Refs#

  • airflow.apache.org docs
  • dagster.io docs
  • temporal.io docs

FAQ#

How does a data pipeline orchestrator work?#

Developers define DAGs of tasks in code. A scheduler determines which tasks are runnable, dispatches them to workers, tracks dependencies, retries failures, and exposes run status.

What is a DAG in a workflow orchestrator?#

A directed acyclic graph defines task dependencies. The orchestrator runs each task only after its parents succeed and stops or retries on failure based on policy.

How does Airflow handle backfills?#

Backfill replays a DAG over historical date ranges. The scheduler creates one run per logical date and executes tasks idempotently with the templated execution date.

Why use Airflow vs Dagster vs Prefect?#

Airflow is mature with a huge plugin ecosystem. Dagster offers strong typing and asset-based modeling. Prefect emphasizes dynamic workflows with hybrid execution.

How is observability achieved in data pipelines?#

Orchestrators expose task logs, runtime metrics, SLA misses, and lineage. Integrations with tools like OpenLineage track upstream sources and downstream consumers.

How do orchestrators scale workers?#

Schedulers dispatch tasks to a pool of executors like Celery, Kubernetes, or local processes, autoscaling worker count based on queue depth and resource requests.

Further reading#