Skip to content

CI/CD System#

Problem statement (interviewer prompt)

Design a CI/CD platform like GitHub Actions or Jenkins. Receive webhooks from a VCS, schedule pipelines defined in YAML, run jobs on ephemeral runners across many environments, surface logs in real-time, and gate deploys with manual approvals.

Continuous delivery pipeline: build, automated tests, and staged deploys from commit to production
Source: Wikimedia Commons. CC BY-SA 4.0.
flowchart TB
  VCS[(Git server)] -->|webhook on push| Ingest[Webhook ingest]
  Ingest --> PQ[(Pipeline queue)]
  PQ --> Sched[Scheduler]
  Sched --> Runner1[Ephemeral runner 1]
  Sched --> Runner2[Ephemeral runner 2]
  Sched --> RunnerN[Runner N]
  Runner1 --> LogBus[(Log stream Kafka)]
  Runner1 --> Art[(Artifact store)]
  LogBus --> UI[Live log UI]
  Runner1 --> Deploy[Deploy stage]
  Deploy --> Approve{Manual approval}
  Approve -->|approved| Prod[(Production)]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class VCS,Prod datastore;
    class Ingest,Sched,Deploy,UI service;
    class Runner1,Runner2,RunnerN compute;
    class PQ,LogBus queue;
    class Art storage;

Pipeline as code (YAML); ephemeral runners (containers or VMs); log streaming; per-environment secrets; approval gates.

A CI/CD platform translates "push to main" into "tested artifact in production" with audit trails, approvals, and rollback.

Pipeline DAG#

flowchart TB
  Build --> Test1[Unit tests]
  Build --> Test2[Lint]
  Build --> Test3[Integration tests]
  Test1 --> Pkg[Package artifact]
  Test2 --> Pkg
  Test3 --> Pkg
  Pkg --> Stage[Deploy staging]
  Stage --> E2E[E2E tests]
  E2E --> Gate{Approval}
  Gate -->|approved| Prod[Deploy prod canary]
  Prod --> Mon[Monitor SLO]
  Mon -->|ok| Promote[Promote 100%]
  Mon -->|fail| Roll[Auto-rollback]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Build,Test1,Test2,Test3,Pkg,Stage,E2E,Prod,Mon,Promote,Roll service;
    class Gate edge;

Runner pool#

Type Use
Self-hosted persistent Cheap; security risk if shared
Self-hosted ephemeral (autoscaled K8s) Default; clean state each run
Cloud-hosted (GitHub-hosted) Convenient; cost-effective for low scale
Spot / preemptible Cheap; needs retry on eviction

Schedule jobs by labels (linux-x64, gpu, windows). Each runner pulls one job, runs in fresh container, exits.

Caching#

Massive speed-up. Caches per pipeline / branch / org:

  • Dependency cache (npm, pip, gradle, cargo)
  • Docker layer cache (BuildKit + S3 backend)
  • Compiler output (sccache, ccache)
  • Test results (skip unchanged tests)

Cache key derived from lockfile hash; eviction by LRU.

Log streaming#

sequenceDiagram
  participant R as Runner
  participant LB as Log bus (Kafka)
  participant UI as Browser
  R->>LB: stdout/stderr line (job_id, ts)
  LB->>UI: tail (WebSocket)
  R->>LB: completion
  R->>S3: upload full log archive

UI shows live tail; archived log searchable later (Elasticsearch / Loki).

Secrets#

  • Stored encrypted at rest (KMS).
  • Injected as env vars only to authorised pipelines.
  • Per-environment scoping (staging ≠ prod).
  • OIDC-based cloud auth: runner gets short-lived AWS/GCP credentials via federation - no long-lived keys.

Deployment gates#

Gate When
Required reviewers Risky envs (prod, regulated)
Cool-down between deploys Bake time for canary
Time window "No prod deploys Friday after 4pm"
SLO budget check Block deploy if error budget exhausted
Schema migration lock Coordinate DDL across services

Self-hosted runner security#

A pipeline runs untrusted code from the repo. Mitigations:

  • Run in a fresh container, never on host.
  • No host volume mounts.
  • Network egress allowlist.
  • Cache contents from untrusted PRs are NOT shared with main.

Capacity sketch#

10k engineers × 30 builds/day = 300k pipelines/day; avg 5 jobs × 2 min = 50k runner-hours/day = ~2000 concurrent runners.

Tools#

Tool Notes
GitHub Actions Cloud + self-hosted
GitLab CI Same
Jenkins Old; pipeline-as-code via Jenkinsfile
CircleCI / BuildKite SaaS
Argo Workflows K8s-native
Tekton K8s-native, low-level
Drone Lightweight

How CI/CD composes#

flowchart TB
  CICD((CI/CD<br/>system))
  DS[Deployment strategies<br/>orchestrated at deploy stage]
  CO[Container orchestration<br/>runner substrate]
  CR[Container registry<br/>artifact target]
  SEC[Security fundamentals<br/>secrets injection]
  CICD --> DS
  CO --> CICD
  CICD --> CR
  SEC --> CICD

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class CICD service;
    class DS,CO,CR,SEC datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Deployment strategies what the pipeline orchestrates deployment-strategies
HLD Container Orchestration runner substrate container-orchestration
HLD Container Registry artifact destination container-registry
HLD Secrets / KMS injected credentials security-fundamentals

Quick reference#

Functional#

  • VCS webhook → pipeline trigger
  • YAML pipeline definition (DAG of jobs)
  • Ephemeral runner pool
  • Live log stream + archive
  • Artifact build + registry push
  • Deploy with approval gates + rollback

Non-functional#

  • Pipeline start latency < 30s
  • Concurrent capacity 1000s of runners
  • Per-tenant isolation
  • Audit trail for compliance

Components#

  • Webhook ingest
  • Pipeline scheduler
  • Runner controller (autoscale K8s)
  • Log bus (Kafka) + archiver (S3 + ES/Loki)
  • Artifact store (container reg, OCI artifacts)
  • Secrets store (KMS-backed, OIDC for cloud)
  • Approval engine

Caching layers#

  • Dependency (npm, pip, gradle)
  • Docker layer (BuildKit + S3)
  • Compiler (sccache, ccache)
  • Test selection (skip unchanged)

Security#

  • Ephemeral container per job
  • Egress allowlist
  • No PR cache poisoning into main
  • OIDC for short-lived cloud creds

Deploy gates#

Reviewers, time windows, SLO budget, cool-down, schema-migration lock.

Capacity sketch#

10k engineers × 30 builds × 5 jobs × 2 min ≈ 2000 concurrent runners.

Tools#

GitHub Actions, GitLab CI, Jenkins, Argo Workflows, Tekton, CircleCI, BuildKite, Drone.

Refs#

  • GitHub Actions docs
  • Argo Workflows
  • Google - Modern CD

FAQ#

How does a CI/CD system work?#

A CI/CD system listens for VCS webhooks, parses pipeline YAML, schedules jobs onto ephemeral runners, captures logs and artifacts, and optionally promotes builds through deploy gates.

What is the difference between CI and CD?#

Continuous Integration builds and tests every change. Continuous Delivery automates packaging and staging deploys, while Continuous Deployment auto-promotes to production.

Why do CI/CD systems use ephemeral runners?#

Ephemeral runners give each job a clean isolated environment, eliminate state leakage between builds, and scale elastically with workload spikes.

How are secrets handled in a CI/CD pipeline?#

Secrets are stored encrypted in a vault, injected as environment variables at run time, masked in logs, and scoped per environment so untrusted PRs cannot exfiltrate them.

What is the difference between blue-green and canary deployment?#

Blue-green swaps two full environments. Canary routes a small percentage of traffic to the new version, monitors metrics, and gradually ramps up before full rollout.

How do CI systems show real-time build logs?#

Runners stream stdout via a long-lived connection to a log service, which fans out chunks to the UI via WebSockets so users see output as it happens.

Further reading#