Load Balancer#

Q: What is a load balancer?

A load balancer fronts a pool of backend servers behind one virtual endpoint, spreads requests across healthy backends, and steers traffic away from failing nodes.

Q: Layer 4 vs layer 7 load balancer?

L4 routes by IP and port and is faster and cheaper. L7 understands HTTP, can route by path, header, and cookie, and is needed for advanced traffic shaping.

Q: How do load balancer health checks work?

Active health checks probe each backend on an interval and remove failing instances. Passive checks track real request errors and eject instances that breach thresholds.

Q: What is consistent hashing in load balancing?

Consistent hashing maps each key to a backend on a hash ring, so adding or removing a backend only reassigns 1/N of keys, preserving caches and session affinity.

Q: Round robin vs least connections, which is better?

Round robin works for uniform requests. Least connections handles bursty or long-running calls better, avoiding pile-up on slow backends with stuck connections.

Problem statement (interviewer prompt)

Design a layer that fronts a pool of identical backend servers. Clients send to one virtual endpoint; the layer must distribute traffic, detect and bypass unhealthy nodes within seconds, support multiple algorithms, and survive its own host failures. Aim for sub-millisecond added latency at 100k+ RPS.

A load balancer distributes traffic across a pool of backend servers, providing horizontal scale, failover and a single virtual endpoint to clients.

flowchart LR
  C1([Client A])
  C2([Client B])
  C3([Client C])
  LB{{Load Balancer<br/>VIP}}
  S1[Server 1]
  S2[Server 2]
  S3[Server 3]
  C1 --> LB
  C2 --> LB
  C3 --> LB
  LB --> S1
  LB --> S2
  LB --> S3
  S1 -.health.-> LB
  S2 -.health.-> LB
  S3 -.health.-> LB

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C1,C2,C3 client;
    class LB edge;
    class S1,S2,S3 service;

flowchart TB
  subgraph Client[Clients]
    C1([Browser])
    C2([Mobile App])
    C3[Service-to-service]
  end

  subgraph DNS_Tier[DNS / GSLB]
    DNS[Authoritative DNS]
    GSLB[Geo / Latency<br/>Routing - Route53 / Akamai]
  end

  subgraph Edge[Edge L4/L7 Tier]
    Anycast[Anycast IP / BGP]
    L4[L4 LB<br/>ECMP, Maglev,<br/>LVS / IPVS]
    L7A[L7 LB A<br/>Envoy / NGINX / HAProxy]
    L7B[L7 LB B<br/>Active]
    VRRP[(VRRP / Keepalived<br/>HA pair)]
  end

  subgraph Algos[Selection Algorithms]
    RR[Round Robin]
    WRR[Weighted RR]
    LC[Least Connections]
    EWMA[EWMA Latency]
    PEAK[Power-of-2-Choices]
    HASH[Consistent Hash<br/>session/sticky]
  end

  subgraph Health[Health & Service Discovery]
    HC[Active Health Checks<br/>HTTP/TCP/gRPC]
    PC[Passive checks<br/>outlier detection]
    SD[Service Registry<br/>Consul / etcd / xDS]
  end

  subgraph Pool[Backend Pool]
    direction LR
    B1[Backend 1]
    B2[Backend 2]
    B3[Backend 3]
    BN[Backend N]
  end

  subgraph Observability
    M[(Metrics<br/>p50/p95/p99, RPS)]
    L[(Access Logs)]
    T[(Traces)]
  end

  C1 --> DNS
  C2 --> DNS
  C3 --> DNS
  DNS --> GSLB
  GSLB --> Anycast
  Anycast --> L4
  L4 --> L7A
  L4 --> L7B
  L7A <-.VRRP failover.-> L7B
  L7A --> Algos
  Algos --> Pool
  SD -.config push.-> L7A
  SD -.config push.-> L7B
  HC --> B1
  HC --> B2
  HC --> B3
  HC --> BN
  B1 -.status.-> SD
  B2 -.status.-> SD
  B3 -.status.-> SD
  BN -.status.-> SD
  L7A --> M
  L7A --> L
  L7A --> T
  PC -. eject on 5xx .-> L7A

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C1,C2 client;
    class DNS,Anycast,L4,L7A,L7B edge;
    class C3,GSLB,RR,WRR,LC,EWMA,PEAK,HASH,HC,PC,SD,B1,B2,B3,BN service;
    class VRRP,M,L,T datastore;

Notes#

L4 vs L7: L4 forwards TCP/UDP (fast, opaque); L7 understands HTTP/gRPC, can do path routing, retries, header manipulation, TLS termination, mTLS.
TLS: terminate at L7; re-encrypt to backend if zero-trust required.
HA: keepalived/VRRP for active-passive, or anycast + ECMP for active-active (Google Maglev, Cloudflare Unimog).
Sticky sessions: cookie-based (SERVERID) or source-IP hash; prefer stateless tokens.
Outlier detection: eject hosts on consecutive 5xx; gradually re-admit.
Rate limiting & circuit breaking are commonly co-located at L7.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Load balancer / GSLB	L4/L7 traffic distribution and failover	load-balancer
`HLD`	CDN	edge caching for static assets	cdn
`HLD`	Consistent hashing	key placement with minimal remap	consistent-hashing
`HLD`	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`HLD`	Service mesh	sidecar mesh, mTLS, traffic policy	service-mesh
`HLD`	Multi-region & DR	RTO / RPO, active-active, failover	multi-region-dr

Quick reference#

Functional requirements#

Distribute incoming traffic across N backends.
Detect and bypass unhealthy backends within seconds.
Support multiple algorithms (RR, WRR, least-conn, consistent hash).
Terminate TLS (optional) and route by host/path (L7).

Non-functional requirements#

Throughput: 1M+ RPS per LB pair (commodity NIC).
Latency overhead: < 1 ms p99 added by LB.
Availability: 99.99%+. No single LB = SPOF.
Horizontal scale: ECMP + multiple LB nodes.

Capacity estimation (example)#

100k RPS, 1 KB request, 10 KB response → ~10 Gbps egress.
Connections: 100k RPS × 0.5 s avg keep-alive = 50k concurrent.
File descriptors per LB box: 200k+ (tune ulimit, ephemeral ports).

API surface#

Control plane: add_backend(host, weight), drain(host), set_health(host).
Data plane: transparent - clients send to VIP, LB forwards.
xDS (Envoy) for dynamic config push.

Data model#

Pool{ id, algo, hc_config }
Backend{ id, pool_id, addr, weight, state(UP/DOWN/DRAIN) }
Listener{ vip, port, tls_cert, route_rules[] }

Trade-offs#

L4 = cheapest, fastest, opaque to app; L7 = richer features, ~2-5× CPU.
DNS LB = simple but slow failover (TTL); GSLB needed for multi-region.
Sticky sessions simplify legacy apps but pin load; prefer stateless JWT.
Active-active anycast scales out but requires BGP; VRRP active-passive is easier ops.
TLS termination at edge improves CPU on backend but exposes plaintext in DC; mTLS to backend solves it at cost.

Real-world refs#

Google Maglev (consistent hash + ECMP), Facebook Katran (XDP/eBPF L4), AWS NLB (L4) / ALB (L7), Cloudflare Unimog, Envoy + Istio.

FAQ#

What is a load balancer?#

A load balancer fronts a pool of backend servers behind one virtual endpoint, spreads requests across healthy backends, and steers traffic away from failing nodes.

Layer 4 vs layer 7 load balancer?#

L4 routes by IP and port and is faster and cheaper. L7 understands HTTP, can route by path, header, and cookie, and is needed for advanced traffic shaping.

How do load balancer health checks work?#