Skip to content

Health Check / Heartbeat Service#

Problem statement (interviewer prompt)

Design a health-check / heartbeat service used by thousands of microservices for liveness / readiness probes + dashboards. Aggregate at 1Hz, detect failures within seconds, distinguish flap vs hard-down, and feed into the LB / orchestrator's removal decisions.

flowchart LR
  N[Nodes]
  HB[Heartbeat collector]
  STATE[(Health state)]
  ALERT[Alerts]
  N -. heartbeat .-> HB --> STATE --> ALERT

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class N,HB service;
    class STATE datastore;
    class ALERT obs;

flowchart TB
  subgraph Nodes
    N1[Node 1]
    N2[Node 2]
    NN[Node N]
  end

  subgraph Modes[Probe modes]
    ACTIVE[Active checks - HTTP/TCP/gRPC]
    PASSIVE[Passive - outcome-based]
    HB[Heartbeat push from node]
    PHI[Phi-accrual detector]
  end

  subgraph Service
    COL[Collector cluster]
    STATE[(Health state KV)]
    GOSSIP[Gossip across collectors]
    RULES[Status rules]
    DEPS[Dependency graph]
  end

  subgraph Reactions
    LB[LB pool update]
    ROUTE[Routing change]
    ALERT[Alerts / pages]
    RUNBOOK[Automated remediation]
  end

  Nodes --> Modes --> Service --> Reactions

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class LB edge;
    class N1,N2,NN,ACTIVE,PASSIVE,HB,PHI,COL,GOSSIP,RULES,DEPS,ROUTE,RUNBOOK service;
    class STATE datastore;
    class ALERT obs;

Failure detection#

  • Binary up/down is brittle.
  • Phi-accrual outputs a suspicion value (probability host is dead); thresholding triggers reactions.

Probe vs heartbeat#

  • Active probe = control plane initiates; ensures network path works.
  • Heartbeat = node-initiated; cheaper at scale.
  • Both used together.

Glossary & fundamentals#

Concept What it is Fundamentals
Service discovery feeds healthy set service-discovery
Load balancer consumes healthy pool load-balancer
Resilience patterns breakers + ejections resilience-patterns
Observability uptime SLO signals observability

Quick reference#

Functional#

  • Periodic active and/or passive checks.
  • Distributed agreement on liveness.
  • Feed routing + LB + alerts.
  • Dependency-aware (don't page if upstream is the real cause).

Non-functional#

  • Sub-second detection ideal.
  • Avoid flapping (debounce, hysteresis).
  • Survive network partitions safely (prefer caution).

Trade-offs#

  • Frequent probes = fast detection but load.
  • Phi accrual = nuanced; tuning effort.
  • Multi-checker quorum to avoid single observer triggering global change.

Refs#

  • Cassandra & Akka phi-accrual.
  • Consul / etcd / ZK health docs.
  • "The Tail at Scale" Dean & Barroso.

FAQ#

What is a heartbeat service?#

A heartbeat service collects periodic liveness signals from nodes, stores recent samples, and emits alerts when a node misses N consecutive intervals. Orchestrators consume its verdict for removal decisions.

Active vs passive health checks?#

Active checks have a central prober dial each node on a schedule. Passive checks let each node push its own heartbeat. Active is simpler; passive scales better since the load is on the node, not the prober.

Liveness probe vs readiness probe?#

Liveness asks 'is the process alive'; if it fails, the orchestrator restarts the pod. Readiness asks 'is it ready to take traffic'; if it fails, the load balancer just stops sending requests.

How do you avoid false positives from flapping?#

Require N consecutive misses before marking down, and N consecutive successes before marking back up. Add jitter to probe schedules so a network blip does not synchronize failures.

How does the health service alert at scale?#

Aggregate state in memory per region, write durable transitions to a log, and let an alerting service tail the log. This separates fast detection from heavyweight notification delivery.