Skip to content

Chaos Engineering#

Problem statement (interviewer prompt)

A distributed system claims to tolerate single-AZ failure, primary database failover, and dependency timeouts. How do you verify those claims hold continuously, before a real outage exposes the gap?

Chaos engineering treats resilience as a property to be continuously verified by injecting failures and observing system behaviour. Hypothesise; inject; measure; learn.

flowchart LR
  H[Hypothesis<br/>system tolerates X] --> Inj[Inject failure<br/>kill pod / drop network]
  Inj --> Obs[Observe<br/>SLOs hold?]
  Obs -->|yes| Doc[Document]
  Obs -->|no| Fix[Find weakness, fix]
  Fix --> H

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class H,Inj,Obs,Fix,Doc service;

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;

Netflix's Chaos Monkey randomly kills VMs in production. Modern tools (Litmus, Chaos Mesh, AWS FIS, Gremlin) inject CPU stress, latency, DNS failure, dependency timeouts, and pod kills with safety guards.

Chaos engineering, formalised at Netflix in 2010, is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production.

Five principles#

  1. Build a hypothesis around steady-state behaviour. What does normal look like? Pick measurable SLIs.
  2. Vary real-world events. Hardware failures, malformed responses, latency spikes, traffic shifts.
  3. Run experiments in production. Staging never captures production scale or interactions.
  4. Automate to run continuously. A one-time game day is theatre.
  5. Minimise blast radius. Smallest experiment that proves the hypothesis.

Experiment lifecycle#

sequenceDiagram
  participant E as Engineer
  participant P as Chaos platform
  participant Sys as Production
  participant Mon as Monitors
  E->>P: define experiment (target, fault, duration)
  P->>Mon: snapshot baseline SLOs
  P->>Sys: inject fault (limited scope)
  Sys-->>Mon: emit metrics
  Mon-->>P: SLO budget check
  alt SLO holds
    P-->>E: pass, document
  else SLO violated
    P->>Sys: abort, restore
    P-->>E: fail, weakness found
  end

Safety gates: dry run in staging first, blast-radius limits, automatic abort on SLO breach, business-hours-only by default.

Fault catalog#

Layer Faults
Infrastructure VM/pod kill, AZ isolation, region failover
Network Latency, packet loss, DNS poison, partition
Compute CPU stress, memory pressure, disk fill
Application HTTP 5xx injection, timeout, malformed payload
Dependency Database failover, queue backlog, cache eviction
Time Clock skew, NTP drift

Netflix Simian Army#

Tool Failure
Chaos Monkey Random instance termination
Latency Monkey Inject HTTP/gRPC latency
Conformity Monkey Find instances not meeting best practice
Janitor Monkey Clean unused resources
Chaos Gorilla Take down an entire AZ
Chaos Kong Take down an entire region

GameDays#

A GameDay is a scheduled, human-driven chaos exercise:

flowchart TB
  Plan[Plan: hypothesis, target] --> Brief[Brief: on-call notified]
  Brief --> Inject[Inject: at scheduled time]
  Inject --> Watch[Observe + react]
  Watch --> Retro[Retro: did detection/response work?]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;

GameDays test the humans as much as the system: was the runbook accurate? Did alerts fire? Did the right team get paged?

Common findings#

  • Timeouts are too long; one slow dependency stalls every thread.
  • Retries lack jitter and amplify the problem (retry storm).
  • Circuit breakers exist but were never tested open.
  • Health checks lie - the pod is "ready" but its dependency is down.
  • Dashboards exist but no one notices the regression.

Integration with deploys#

Run chaos experiments after canary promotion - if a new deploy weakens resilience, fail the rollout:

flowchart LR
  Deploy --> Canary --> Promote[Promote 25%]
  Promote --> Chaos[Run chaos test]
  Chaos -->|pass| Promote100[100%]
  Chaos -->|fail| Rollback

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;

Tools#

Tool Scope
Chaos Mesh Kubernetes-native, broad fault catalog
Litmus K8s, CNCF
AWS Fault Injection Service AWS resources
Gremlin SaaS, supports any infra
Toxiproxy Network failure proxy
Pumba Docker chaos

Where chaos engineering connects#

flowchart TB
  CE((Chaos<br/>engineering))
  RES[Resilience patterns<br/>verifies controls work]
  OBS[Observability<br/>SLO budget signal]
  DS[Deployment strategies<br/>gate at promotion]
  MR[Multi-region DR<br/>highest-stakes target]
  RES --> CE
  OBS --> CE
  DS --> CE
  CE --> MR

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class CE service;
    class RES,OBS,DS,MR datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Resilience patterns timeouts, retries, circuit breakers chaos tests resilience-patterns
HLD Observability the measurement substrate observability
HLD Deployment strategies gates chaos tests at promotion deployment-strategies
HLD Multi region DR tested via region chaos multi-region-dr

Quick reference#

Principles#

  1. Steady-state hypothesis
  2. Vary real-world events
  3. Run in production
  4. Continuous, automated
  5. Minimal blast radius

Experiment recipe#

1. Snapshot SLOs
2. Inject scoped fault
3. Watch SLO budget
4. Abort on breach
5. Document outcome

Fault catalog#

Layer Faults
Infra Pod kill, AZ down
Network Latency, drop, partition
Compute CPU, mem, disk stress
App 5xx, timeout, malformed
Dep DB failover, queue backlog
Time Clock skew

Safety gates#

  • Dry-run in staging
  • Abort on SLO burn
  • Blast radius < 5% by default
  • Business hours, on-call notified

Tools#

Tool Note
Chaos Mesh K8s native
Litmus K8s, CNCF
AWS FIS AWS-native
Gremlin SaaS, any infra
Toxiproxy Network proxy

Common findings#

  • Timeouts too long → thread starvation
  • Retries without jitter → storms
  • Untested circuit breakers
  • Health checks lie
  • Alerts fire but no one notices

Refs#

  • principlesofchaos.org
  • Netflix Tech Blog - Simian Army
  • Rosenthal & Jones - Chaos Engineering book

FAQ#

What is chaos engineering?#

Chaos engineering is the discipline of deliberately injecting failures into a system to verify its resilience claims. The goal is to find weaknesses before a real outage exposes them.

Is chaos engineering safe in production?#

Yes when scoped carefully with a small blast radius, a clear hypothesis, automatic abort criteria, and steady-state metrics. Start in staging, then small percentages of production traffic.

What is Chaos Monkey?#

Chaos Monkey is a Netflix tool that randomly terminates production instances to ensure services tolerate node failure. It was the first widely known chaos engineering tool and inspired the field.

What are the steps of a chaos experiment?#

Define steady state, form a hypothesis, choose a failure to inject, run the experiment with a blast radius limit, observe metrics, and either confirm the hypothesis or document a fix.

Gremlin, Chaos Mesh, LitmusChaos, AWS Fault Injection Simulator, and Azure Chaos Studio are popular. Each supports pod kills, latency injection, network drops, and CPU stress.

Further reading#