Skip to content

Multi-Region & Disaster Recovery#

Problem statement (interviewer prompt)

Make a system survive losing an entire AWS region. The product has a 5-minute RTO and 30-second RPO. Choose between cold standby, pilot light, warm standby, active-passive, and active-active. Cover traffic steering, data replication, and game-day testing.

Concept illustration 
flowchart LR
  U[Users worldwide]
  R1[Region A]
  R2[Region B]
  R3[Region C]
  GLB[Geo / latency LB]
  U --> GLB
  GLB --> R1
  GLB --> R2
  GLB --> R3
  R1 <-. async replication .-> R2
  R2 <-. async replication .-> R3

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class GLB edge;
    class U,R1,R2,R3 service;

Make your system survive losing an entire region. Two axes: RTO (how fast can you recover?) and RPO (how much data can you lose?). Patterns range from cold standby (high RTO, high RPO) to active-active (RTO ≈ 0, RPO ≈ 0).

Patterns#

flowchart TB
  subgraph Cold[Cold standby]
    C1[Primary serves all]
    C2[(Backups only)]
    C3[RTO: hours-days, RPO: minutes-hours]
  end
  subgraph Pilot[Pilot light]
    P1[Primary serves all]
    P2[Standby idle, DB replicated]
    P3[RTO: tens of minutes, RPO: minutes]
  end
  subgraph Warm[Warm standby]
    W1[Primary serves all]
    W2[Standby running, scaled down]
    W3[RTO: minutes, RPO: seconds]
  end
  subgraph ActPas[Active-passive]
    AP1[Primary serves all]
    AP2[Standby at full size]
    AP3[RTO: seconds-1m, RPO: seconds]
  end
  subgraph ActAct[Active-active]
    AA1[Both regions serve]
    AA2[Conflict resolution]
    AA3[RTO ≈ 0, RPO ≈ 0]
  end

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C1,C3,P1,P3,W1,W2,W3,AP1,AP2,AP3,AA1,AA2,AA3 service;
    class C2,P2 datastore;

Concepts#

Term Meaning
RTO Recovery time - how long until traffic flows again
RPO Recovery point - how recent the last good data is
MTBF Mean time between failures
MTTR Mean time to repair
Failover Switch traffic to standby on failure
Failback Move traffic back once primary is healthy
Failover region Where traffic goes when primary dies

Traffic steering#

flowchart LR
  DNS[Geo / latency DNS<br/>or anycast]
  HC[Health checks per region]
  R1[Region A]
  R2[Region B]
  R3[Region C]
  DNS --> HC
  HC -.->|A unhealthy| DNS
  DNS --> R1
  DNS --> R2
  DNS --> R3

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class DNS edge;
    class HC,R1,R2,R3 service;
  • GeoDNS - short TTL (30-60s), routes by client geography.
  • Anycast - same IP advertised from multiple POPs; BGP picks closest.
  • Global load balancer - Cloudflare, AWS Global Accelerator, GCP GLB.

Data layer#

Layer Active-active option Trade-off
Stateless services trivial none
Cache per-region (Redis cluster) or hierarchical cross-region inval
OLTP DB Spanner, CockroachDB, Aurora Global, DynamoDB Global latency on cross-region writes
Object store S3 cross-region replication eventual; pay for replication
Event bus MirrorMaker (Kafka), cross-region Pub/Sub dedupe at consumer

Cross-region write strategies#

  • Region-pinned writes - each user writes to "home region", reads global. Easiest.
  • Last-writer-wins - eventual; data loss possible on conflict.
  • CRDTs / per-key conflict resolution - strong eventual consistency for counters, sets.
  • Globally consistent (Spanner / CRDB) - TrueTime / Raft over WAN; ~150ms commit.

Game days#

  • Practice failover quarterly.
  • Tag every infra resource with failover_role.
  • Document the runbook; chaos-test it.

DR levels (AWS Well-Architected)#

flowchart LR
  L1[Backup & Restore]
  L2[Pilot Light]
  L3[Warm Standby]
  L4[Active-Active / Multi-Site]
  L1 --> L2 --> L3 --> L4
  L1 -. cheaper, slower .- L1
  L4 -. more expensive, faster .- L4

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class L2,L3,L4 service;
    class L1 datastore;

Common interview hooks#

  • "What's the latency cost of active-active for writes?" → cross-region RTT; bounded by physics.
  • "How do you handle stateful workloads in failover?" → drain, replicate, promote, fence (STONITH).
  • "Active-active how to avoid split-brain?" → leases, fencing tokens, quorum across regions.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD Load balancer / GSLB L4/L7 traffic distribution and failover load-balancer
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD CAP / PACELC C vs A under partition; L vs C otherwise cap-pacelc
HLD Raft / Paxos consensus replicated state machine via majority quorum consensus-raft-paxos
HLD Leader/follower replication sync/semi-sync/async replication, failover replication-leader-follower
HLD Logical clocks Lamport, vector clocks, HLC, TrueTime logical-clocks
HLD Multi-region & DR RTO / RPO, active-active, failover multi-region-dr

Quick reference#

Setting RTO / RPO#

  • Talk to the business - "5 min RTO" is a different cost from "5 hr RTO".
  • Tier services: tier-0 (payments, login) → active-active; tier-3 (analytics) → backups OK.

Cost#

  • Active-active doubles compute + state replication bills.
  • Cross-region egress is the silent killer. Estimate before you commit.

Stateful service failover steps#

  1. Detect (health checks across regions, plus quorum check).
  2. Fence the old primary (STONITH / lease revocation).
  3. Promote standby (or new region) to primary.
  4. Repoint DNS / GLB to new region (or shift weight gradually).
  5. Drain caches / warm them up.
  6. Failback later when source region is healthy.

Avoid these antipatterns#

  • "Failover plan that's never been tested" - equals no plan.
  • Hidden single-region dependencies (KMS keys, secret store, image registry).
  • Long-TTL DNS - clients keep hammering the dead region.
  • Asymmetric failover (DB fails over but service config doesn't).

Refs#

  • AWS Well-Architected Reliability Pillar (DR strategies).
  • "Site Reliability Engineering" book - chapter on failure recovery.
  • Spanner / CockroachDB papers on geo-replication.
  • Netflix Chaos Kong + RegionalEvac runbooks.

FAQ#

What is RTO vs RPO?#

RTO is recovery time objective, how fast service comes back. RPO is recovery point objective, how much recent data you accept losing. Both drive DR design choices.

Active-active vs active-passive, when use which?#

Active-active gives the lowest RTO but needs conflict resolution and double the cost. Active-passive is simpler and cheaper, fine when minutes of failover are acceptable.

What is pilot light architecture?#

Pilot light keeps a minimal stack running in the standby region with data replicated continuously. On disaster you scale the rest of the stack up, trading cost for RTO.

How does cross-region data replication work?#

Async log shipping (MySQL binlog, Postgres WAL) covers most stores. Globally synchronous DBs like Spanner and CockroachDB use Paxos or Raft across regions for strong consistency.

Why are DR game days important?#

Untested failover plans usually fail. Regular game days expose stale DNS records, broken IAM, missing capacity, and runbook gaps, the things that turn an outage into a disaster.

Further reading#

Curated, high-credibility sources for going deeper on this topic.