Skip to content

Deployment Strategies#

Problem statement (interviewer prompt)

A service running v1 must roll out v2 to thousands of pods across multiple regions. Rollouts must be safe, observable, and rapidly reversible if the error budget burns. Compare the standard strategies and pick the right one per workload.

Five patterns differ on how many users see the new version at any moment, and how fast you can roll back:

flowchart LR
  subgraph BG[Blue/Green]
    BGL[LB] -->|100%| Blue[v1 fleet]
    BGL -.cutover.-> Green[v2 fleet]
  end
  subgraph Canary
    CL[LB] -->|99%| C1[v1]
    CL -->|1%| C2[v2]
  end
  subgraph Rolling
    R[v1 + v1 + v1 + v1] --> R2[v1 + v1 + v1 + v2] --> R3[v1 + v1 + v2 + v2]
  end

    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class BGL,CL edge;
    class Blue,Green,C1,C2 service;

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;

Choose by risk budget, rollback speed, and capacity overhead.

A deployment strategy answers three questions:

  1. Blast radius: how many users hit v2 in the worst case?
  2. Rollback time: how quickly can we revert if v2 is bad?
  3. Cost: how much extra capacity do we pay during rollout?

Blue/Green#

flowchart TB
  LB[Load balancer] -->|100%| B[Blue v1 fleet]
  LB -.cutover.-> G[Green v2 fleet]
  B -. on rollback .-> B

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
  • Stand up an entire v2 fleet in parallel ("green"), warm caches, smoke-test.
  • Flip the load balancer atomically.
  • Rollback = flip back; near-instant.
  • Capacity: 2x during cutover. Expensive at large fleets.

Canary#

flowchart LR
  LB --> CV1[v1: 99%]
  LB --> CV2[v2: 1%]
  CV2 -. metrics ok? .-> Promote[promote to 5% → 25% → 100%]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
  • Send a small slice of traffic to v2; measure error rate, latency, business metrics.
  • Increase share in stages: 1% → 5% → 25% → 100% with bake time between.
  • Automated promoter (Argo Rollouts, Flagger, Spinnaker) gates on SLO budget.
  • Best for: most production deploys.

Rolling update#

sequenceDiagram
  participant LB
  participant Set as Pod set
  Set->>Set: pod 1 v1 -> v2 (drain, replace)
  LB->>Set: still routes
  Set->>Set: pod 2 v1 -> v2
  Set->>Set: ... until all pods on v2
  • Replace pods one at a time (or N at a time).
  • Kubernetes Deployment default.
  • Low cost (no parallel fleet), but rollback requires another rolling pass.
  • Cannot easily route by user.

Shadow / mirror#

flowchart LR
  Client --> LB
  LB --> V1[v1: serves response]
  LB -. duplicate .-> V2[v2: discards response]
  V1 --> Client
  V2 --> Metrics[(compare metrics)]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
  • v2 receives a copy of production traffic but its responses are dropped.
  • Used to test performance and correctness on real workloads without exposing users.
  • Watch out: side effects (writes to DB, emails) must be stubbed in v2.

Feature flags / dark launches#

Deploy v2 code everywhere but keep the new behaviour gated:

if feature_flag("new_checkout", user):
    return new_checkout(...)
return old_checkout(...)
  • Decouples deploy from release.
  • Per-user, per-tenant, per-segment targeting.
  • See Feature Flags.

Choosing#

Strategy Best for Rollback Capacity
Blue/Green Stateless services, fast cutover Instant 2x
Canary Production default Within minutes 1.x
Rolling K8s defaults, low-cost Rolling reverse 1.x
Shadow Risky algorithms, perf testing N/A (no traffic exposed) 2x compute, 0 user risk
Feature flag Decoupled release timing Toggle off 1x

Automated rollback signals#

  • SLO budget burn rate (errors per minute > threshold).
  • Latency p99 regression > X%.
  • Business metric drop (conversion, click-through).
  • Crash loop / OOM > N within Y seconds.

Wire to deploy controller (Argo Rollouts, Flagger): on bad signal, halt promotion and revert.

Coordinating database migrations#

Schema changes can't blue/green; they live in shared storage. Standard recipe:

  1. Expand: add nullable column, dual-write.
  2. Migrate: backfill, validate.
  3. Switch reads: app reads from new column.
  4. Contract: drop old column.

Each step must be safe with both v1 and v2 in flight.

How deploy strategies compose#

flowchart TB
  DS((Deployment<br/>strategies))
  FF[Feature flags<br/>runtime dark launch]
  CB[Cell-based architecture<br/>wave deploys per cell]
  CO[Container orchestration<br/>runtime that drives]
  CE[Chaos engineering<br/>verifies rollback]
  DS --> FF
  DS --> CB
  DS --> CO
  CE --> DS

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class DS service;
    class FF,CB,CO,CE datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Feature flags runtime toggle for dark launches feature-flags
HLD Cell-based architecture wave deploys across cells cell-based-architecture
HLD Container orchestration the runtime that drives rollouts container-orchestration
HLD Chaos engineering proactive failure injection chaos-engineering

Quick reference#

Strategy comparison#

Strategy Rollback Capacity Best for
Blue/Green Instant flip 2x Stateless services
Canary Minutes 1.x Default in production
Rolling Reverse rolling 1.x K8s default
Shadow N/A 2x compute Risk-free perf testing
Feature flag Toggle off 1x Decouple deploy/release

Canary progression#

1% → 5% → 25% → 50% → 100% with bake time and SLO gate at each step.

Automated rollback signals#

  • SLO burn rate
  • p99 latency regression
  • Business metric drop (conversion)
  • Crash loop / OOM rate

Tools#

  • Argo Rollouts, Flagger, Spinnaker
  • LaunchDarkly, Unleash, GrowthBook (flags)
  • Kubernetes Deployment + StatefulSet primitives

Schema migration (expand-contract)#

  1. Expand: add column, dual-write
  2. Migrate: backfill
  3. Switch reads
  4. Contract: drop old

Each step must be safe with both versions live.

Refs#

  • Google SRE Book - Release Engineering
  • Fowler - Blue Green Deployment
  • Argo Rollouts / Flagger docs

FAQ#

What are the main deployment strategies?#

Blue/green swaps two full fleets at once, canary sends a small percent to the new version, rolling updates a few pods at a time, shadow replays traffic without serving responses, and feature flags toggle per user.

Blue/green vs canary, what is the difference?#

Blue/green flips 100 percent of traffic to a new fleet instantly, easy to roll back. Canary slowly increases traffic from 1 percent upward, catching problems before full rollout but requiring metrics gating.

What is shadow deployment?#

Shadow (or dark launch) mirrors production traffic to the new version without sending its responses back to users. It catches performance and correctness regressions safely before any user is exposed.

When should I use feature flags?#

Use feature flags when you want to decouple deploy from release, run A/B tests, gradually roll out to internal users or geographies, or kill switch a misbehaving feature without redeploying.

Which deployment strategy is safest?#

Canary plus feature flags is the gold standard: deploy code dark, expose to 1 percent, watch SLOs, then ramp. Blue/green is safest for stateful changes where you cannot mix versions.

Further reading#