Skip to content

Auto-Scaling#

Auto-scaling kubernetes patterns add or remove replicas (or whole nodes) in response to demand so you pay for what you use without falling over when traffic spikes. The three layers most teams run together are HPA for pod count, VPA for pod size, and the Cluster Autoscaler for nodes underneath them.

flowchart TB
  USER[Traffic] --> LB[Load balancer]
  LB --> PODS[Pods]
  HPA[HPA<br/>scales replicas] -- watches metrics --> PODS
  VPA[VPA<br/>rightsizes pods] -- recommends CPU/mem --> PODS
  CA[Cluster Autoscaler<br/>scales nodes] -- adds/removes --> NODES[Node pool]
  PODS -.scheduled on.-> NODES

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  class USER client;
  class LB edge;
  class PODS,NODES compute;
  class HPA,VPA,CA service;

The signal for HPA is usually CPU or memory, but custom metrics (queue depth, RPS, p95 latency) almost always scale better because they reflect what the user actually feels. VPA recommends or applies new resource requests so each pod gets what it needs; combining VPA + HPA on the same metric is dangerous, so usually you pick one axis per workload.

Two timing knobs decide whether scaling helps or hurts: cooldown / stabilization windows prevent the controller from flapping replicas every few seconds, and warm pools / scale-to-zero trade off cold-start latency against idle cost. Predictive scaling adds a schedule on top so the cluster ramps up before the morning rush, not during it.

Problem statement

Your service handles 200 RPS at midnight and 5000 RPS at 9am. Cold starts take 90 seconds. CPU-based HPA fires too late: by the time it spawns pods, p99 latency has already breached the SLO and pages are firing. Design an auto-scaling strategy that absorbs the morning ramp without overprovisioning at night.

Three axes of auto-scaling#

flowchart TB
  subgraph H[Horizontal: replica count]
    HPA[HPA]
    KEDA[KEDA<br/>event-driven]
  end
  subgraph V[Vertical: pod size]
    VPA[VPA<br/>recommender + updater]
  end
  subgraph C[Cluster: node count]
    CA[Cluster Autoscaler]
    KARP[Karpenter<br/>just-in-time provisioning]
  end

  H --> POD[Pods]
  V --> POD
  C --> NODE[Nodes]
  POD -.run on.-> NODE

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class HPA,KEDA,VPA,CA,KARP service;
  class POD,NODE compute;

Horizontal (HPA, KEDA): add or remove replicas. Cheapest to react. Default scaling axis for stateless services.

Vertical (VPA): change CPU and memory requests of existing pods. Useful when load profile changes shape rather than volume. Restart-required for most pods (updateMode: Recreate); careful with stateful sets.

Cluster (Cluster Autoscaler, Karpenter): add or remove nodes. Slowest layer (instance boot + image pull + bootstrap) but the one that bounds cost.

HPA: how it actually decides#

HPA polls metrics every 15 s by default and computes:

desired = ceil(current_replicas * current_metric / target_metric)

If you have 10 pods at 80% CPU and the target is 50%, desired = ceil(10 * 80 / 50) = 16. The controller then enforces the configured min/max and the stabilization window.

sequenceDiagram
  participant M as metrics-server
  participant H as HPA controller
  participant D as Deployment
  participant N as nodes

  loop every 15s
    M->>H: pod CPU = 80%
    H->>H: desired = ceil(10 * 80/50) = 16
    H->>H: stabilization check (no recent scale-down)
    H->>D: set replicas = 16
    D->>N: schedule 6 new pods
  end

Custom metrics let you scale on what matters: RPS, queue depth, in-flight requests. Use the external metrics adapter (Prometheus Adapter) or KEDA which natively supports Kafka lag, SQS depth, Redis list size, etc.

KEDA: event-driven scaling and scale-to-zero#

KEDA wraps HPA with scalers for 60+ event sources and uniquely supports scale-to-zero. For a queue worker:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    name: order-worker
  minReplicaCount: 0      # scale to zero when queue empty
  maxReplicaCount: 100
  cooldownPeriod: 300     # wait 5m of idle before scaling to 0
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1/.../orders
        queueLength: "30"   # one replica per 30 msgs

Scale-to-zero only makes sense if cold-start is acceptable. For a 200 ms latency budget on an API, it usually is not; for batch workers waking up on a queue, it almost always is.

VPA: rightsizing without guessing#

VPA observes resource usage and produces recommendations like "p95 CPU usage over 24h is 250m, requests are set to 1000m". Three modes:

  • Off: only recommends.
  • Initial: applies on pod creation.
  • Auto / Recreate: evicts pods to apply new requests.

Do not combine VPA with HPA on the same metric. If HPA scales on CPU and VPA shrinks pods so their CPU usage rises, HPA scales out, VPA grows them, HPA scales in. Use VPA on memory while HPA scales on CPU, or use VPA in recommend-only mode.

Cluster Autoscaler and Karpenter#

Cluster Autoscaler watches for pending pods and adds nodes from preconfigured node groups when a pod cannot be scheduled. Removes nodes that are underused for 10 minutes by default.

flowchart LR
  P[Pending pod] --> CA[Cluster Autoscaler]
  CA --> ASG[(Node group ASG)]
  ASG --> NODE[New node]
  NODE --> KUBE[kubelet joins]
  KUBE --> POD[Pod scheduled]

  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
  class P,NODE,POD compute;
  class CA,KUBE service;
  class ASG storage;

Karpenter goes further: it picks instance type and zone per pending pod (no preconfigured groups), often using cheaper or larger instances based on actual requests. Faster boot, better bin packing, fewer wasted nodes.

Reactive vs predictive#

Reactive (default HPA / KEDA / CA): observe, then react. Always lags by the metric window plus scale-out time.

Predictive: schedule capacity ahead of known patterns. Two flavors:

  1. Scheduled scaling: hard-coded min replicas on a cron (minReplicas: 50 from 08:30 to 11:00 on weekdays). Simple, works for predictable diurnal traffic.
  2. Model-driven: AWS Predictive Scaling, GCP Recommender. Train a model on historical load, pre-warm before predicted peak. Pairs with reactive HPA for surprise spikes.
flowchart LR
  HIST[Historical traffic] --> MODEL[Forecast model]
  MODEL --> SCHED[Scheduled minReplicas]
  SCHED --> POD[Pre-warmed pods]
  LIVE[Live metrics] --> HPA[Reactive HPA]
  HPA --> POD

  classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class HIST,LIVE obs;
  class MODEL,HPA,SCHED service;
  class POD compute;

Warm pools and pre-pulled images#

For workloads with long boot times (large container images, JVM warmup, model loading), keep a small warm pool:

  • minReplicas above zero so containers are always live.
  • Pre-pulled images on every node (DaemonSet or image cache).
  • For node-level: AWS ASG Warm Pools keep stopped instances ready to start in seconds.
  • Application-level: priming requests, pre-loaded caches, AOT compilation.

Cooldown and hysteresis#

HPA v2 has separate stabilization windows for up and down:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0     # react fast on spikes
    policies:
      - type: Percent
        value: 100                    # double at most
        periodSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300   # wait 5m before scaling down
    policies:
      - type: Percent
        value: 10                     # at most 10% drop per minute
        periodSeconds: 60

Asymmetry is the rule: scale up fast, scale down slow. The cost of being temporarily over-provisioned (dollars) is much less than the cost of being under-provisioned (incidents).

Code: a custom metric HPA#

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 4
  maxReplicas: 200
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 10
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300

This scales when average RPS per pod crosses 100, adds up to 10 pods every 30 seconds, and waits 5 minutes of cool weather before any downscale.

Gotchas#

  • CPU is a bad scaling signal for IO-bound work. A pod waiting on a DB has low CPU but is saturated. Use latency, in-flight requests, or queue depth.
  • HPA and VPA fighting is a real outage shape. Pick one axis or coordinate on different metrics.
  • Cluster Autoscaler is slow. Two to four minutes for a new node. If pods need to schedule now, prewarm headroom (overprovisioning pause pods).
  • Spot interruptions break naive scaling. Spread across instance types and zones; use Karpenter or AWS Spot Allocation Strategy "capacity-optimized".
  • Scale-down storms. Cooldown windows must respect downstream connection-pool resets and in-flight requests. PreStop hooks + terminationGracePeriodSeconds matter.
  • Aggregating metrics over too long a window turns auto-scaling into delayed scaling. Default 5-min averages miss 1-minute spikes.
  • Stateful workloads need different rules. Sharded DBs scale by resharding, not by replicas. HPA does not understand this.

Quick reference#

Three layers#

  • HPA: pod replica count. Polls every 15s. Default scaling axis for stateless workloads.
  • VPA: pod CPU/memory requests. Recommend or evict-and-resize.
  • Cluster Autoscaler / Karpenter: node count. Slow (2 to 4 min) but bounds cost.

HPA formula#

desired = ceil(replicas * current / target) Honors minReplicas, maxReplicas, behavior policies, stabilization windows.

Signals (ranked)#

  1. Custom user-facing metric (RPS, queue depth, in-flight, p95 latency).
  2. Memory (if heap-bound).
  3. CPU (last resort; fine for compute-bound CPU work, bad for IO).

KEDA scalers (event-driven)#

  • Kafka lag, SQS/PubSub depth, Redis lists, Postgres rows.
  • Supports scale-to-zero (cooldownPeriod).
  • Use for batch / async workers where cold start is acceptable.

VPA modes#

  • Off: recommend only.
  • Initial: apply at pod creation.
  • Recreate / Auto: evict to apply.
  • Never combine VPA + HPA on the same metric.

Cluster Autoscaler vs Karpenter#

Aspect CA Karpenter
Node pools Predefined ASGs Just-in-time instance choice
Speed ~2-4 min ~30-60s
Bin packing Pool-bounded Picks cheapest fit
Spot mix Manual Automatic, capacity-optimized

Reactive vs predictive#

  • Reactive: HPA/CA observe then act. Always lags.
  • Scheduled: minReplicas on a cron for known peaks.
  • Predictive (AWS / GCP): model historical, pre-warm; pair with reactive.

Warm pool patterns#

  • minReplicas > 0 for fast paths.
  • ASG Warm Pools: stopped instances ready to start.
  • Pre-pulled images on every node (DaemonSet).
  • Application priming on startup probe.

Cooldown asymmetry#

  • Scale up: fast, aggressive (stabilization 0, double per minute).
  • Scale down: slow (stabilization 5 min, 10% drop per minute).
  • Cost of over-scaling is dollars; under-scaling is incidents.

Anti-patterns#

  • CPU-based HPA on an IO-bound service.
  • HPA and VPA both tuning CPU - flap city.
  • minReplicas: 1 with cold-start time longer than SLO.
  • No PDB during scale-down -> stampede of evictions.
  • maxReplicas huge with no capacity check -> blow up node group budget.
  • Auto-scaling stateful sets with the same rules as stateless deployments.

Operational metrics to watch#

  • hpa_current_replicas, hpa_desired_replicas.
  • Time-to-ready of new pods (cold start).
  • Cluster Autoscaler pending_pods_seconds.
  • Cost per request vs replica count.

Interview probes#

  • Q: "Why is CPU often a bad scaling signal?" A: IO-bound services have low CPU at saturation. Latency or in-flight requests reflect saturation.
  • Q: "How do you handle a 90s cold start?" A: warm pool, predictive / scheduled scaling, pre-pulled images, faster runtime.
  • Q: "HPA vs VPA tradeoff?" A: HPA adds capacity horizontally with no per-pod restart; VPA rightsizes a pod but usually evicts.
  • Q: "What is Karpenter better at?" A: Picks instance type per pending pod, faster, mixes spot/on-demand automatically.

Refs#

  • Kubernetes docs, Horizontal Pod Autoscaler, Vertical Pod Autoscaler (1.30).
  • KEDA docs, Scalers (2024).
  • Karpenter, aws.amazon.com/blogs/containers introductory series.
  • Google SRE Workbook, Managing Load (ch. 11, 2018).
  • AWS, Predictive Scaling for Amazon EC2 Auto Scaling whitepaper.

FAQ#

What is the difference between HPA and VPA?#

HPA scales the number of pod replicas based on metrics like CPU or queue depth. VPA changes the CPU and memory requests of a pod so each one is right-sized. They generally should not run on the same metric.

How does the Kubernetes cluster autoscaler work?#

It watches for pending pods that cannot be scheduled and adds nodes to the cluster from the cloud provider. When nodes are underused for a cool-down window, it drains and removes them.

Can you scale Kubernetes pods to zero?#

Native HPA does not scale below one replica. Tools like KEDA, Knative, and OpenFaaS extend this with event-driven scaling, letting workloads idle at zero and start when a request or message arrives.

What metrics work best for autoscaling?#

Custom metrics that reflect user experience like requests-per-second, queue depth, or p95 latency outperform raw CPU. CPU works for compute-bound jobs but lags on IO-bound services.

Why does my autoscaler thrash?#

Thrash happens when scale-up and scale-down thresholds overlap or cool-down windows are too short. Widen the band, use a longer down-scale stabilization window, and smooth metrics with a moving average.

  • Container Orchestration: Kubernetes is the substrate that HPA, VPA, and Cluster Autoscaler depend on.
  • Capacity Planning: forecasting load to size minReplicas and maxReplicas without going broke.
  • Load Balancer: distributes traffic across the dynamic set of replicas auto-scaling produces.