API Gateway#

Q: What is an API gateway and why use one?

An API gateway is a single entry point that fronts internal microservices and handles TLS termination, authentication, rate limiting, routing, and aggregation, so clients see one consistent API.

Q: What is the difference between an API gateway and a load balancer?

A load balancer distributes traffic across replicas of the same service at L4 or L7. An API gateway adds protocol awareness like auth, rate limiting, request transformation, and per-route routing across many services.

Q: Is an API gateway a single point of failure?

It can be, so production gateways run as a horizontally scaled fleet behind a load balancer with multi-AZ replicas and health checks. The control plane is separated from the data plane to avoid coupling.

Q: What are popular API gateway tools?

Kong, Envoy, AWS API Gateway, Apigee, Tyk, and NGINX with the Plus or OSS modules are common. Cloud-native deployments often pair Envoy or Istio with a thin gateway layer for external traffic.

Q: How does an API gateway handle authentication?

It terminates TLS, validates the bearer token or API key, and forwards a trusted identity header to downstream services. JWT verification and OAuth introspection are the two common flows.

Problem statement (interviewer prompt)

Design a single entry point that fronts dozens of internal microservices for external clients. It must handle TLS termination, authentication, rate limiting, request routing by host/path, response aggregation, and surface a consistent API contract - all with <5ms p99 added latency.

A single entry point that fronts internal services, handling auth, rate limiting and routing.

flowchart LR
  C([Client])
  GW[API Gateway]
  S1([User Service])
  S2[Order Service]
  S3[Payment Service]
  C --> GW
  GW --> S1
  GW --> S2
  GW --> S3

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C,S1 client;
    class GW edge;
    class S2,S3 service;

flowchart TB
  subgraph Clients
    Web([Web SPA])
    Mob([Mobile])
    Part([Partner / B2B])
  end

  subgraph Edge
    DNS[DNS]
    LB[L4/L7 LB]
  end

  subgraph Gateway[API Gateway Cluster]
    direction TB
    TLS[TLS termination]
    AUTHN[AuthN<br/>JWT / OAuth2 / mTLS]
    AUTHZ[AuthZ / RBAC]
    RL[Rate Limiter<br/>token bucket per key]
    QUOTA[Quota / Plans]
    RT[Router<br/>host / path / header]
    TX[Request transform<br/>protocol bridging REST↔gRPC]
    AGG[Response aggregation<br/>BFF / GraphQL]
    CACHE[(Response cache)]
    RETRY[Retry / Timeout /<br/>Circuit breaker]
    WAF[WAF / Bot]
    LOG[Access log + tracing]
  end

  subgraph Control[Control Plane]
    REG[Service Registry<br/>Consul / xDS]
    CFG[Config / Routes]
    SECRETS[Secrets / Keys / JWKS]
    POLICY[Policy Store - OPA]
    AN[Analytics / Billing]
  end

  subgraph Backend[Internal Services]
    S1([User])
    S2[Order]
    S3[Payment]
    S4[Search - gRPC]
    S5[Legacy SOAP]
  end

  Web --> DNS --> LB --> TLS
  Mob --> DNS
  Part --> DNS
  TLS --> WAF --> AUTHN --> AUTHZ --> RL --> QUOTA --> RT
  RT --> TX --> CACHE
  CACHE -->|miss| RETRY
  RETRY --> S1
  RETRY --> S2
  RETRY --> S3
  RETRY --> S4
  RETRY --> S5
  RETRY --> AGG
  AGG --> Web
  REG -.discovery.-> RT
  CFG -.routes.-> RT
  SECRETS -.keys.-> AUTHN
  POLICY -.rules.-> AUTHZ
  Gateway -.metrics.-> AN
  Gateway -.logs.-> LOG

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Web,Mob,Part,S1 client;
    class DNS,LB,TLS,WAF edge;
    class AUTHN,AUTHZ,QUOTA,RT,TX,AGG,RETRY,LOG,REG,CFG,SECRETS,POLICY,AN,S2,S3,S4,S5 service;
    class CACHE cache;
    class RL storage;

Cross-cutting concerns#

AuthN: validate JWT (JWKS rotation), opaque tokens via introspection, mTLS for B2B.
Rate limit: per-API-key, per-IP, per-route; token bucket in Redis.
Transform: REST→gRPC, batch requests, GraphQL gateway / BFF.
Observability: trace ID propagation (W3C traceparent), structured logs.

Deployment#

Active-active cluster behind a L4 LB.
Hot reload of routes (xDS or pull from Git).
Blue/green or canary by route weight.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Load balancer / GSLB	L4/L7 traffic distribution and failover	load-balancer
`HLD`	API gateway / BFF	single ingress, auth, rate limit, routing	api-gateway
`HLD`	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
`HLD`	Resilience patterns	timeout, retry, breaker, bulkhead, backpressure	resilience-patterns
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`HLD`	Service mesh	sidecar mesh, mTLS, traffic policy	service-mesh
`HLD`	Multi-region & DR	RTO / RPO, active-active, failover	multi-region-dr
`LLD`	REST API design	verbs, statuses, pagination, errors	rest-api-design

Quick reference#

Functional requirements#

Single ingress for external traffic.
AuthN/AuthZ, rate limit, quota, request/response transform.
Routing by host/path/header/version.
Aggregation (BFF) for chatty clients.
Protocol bridging (REST ↔ gRPC, WS).

Non-functional#

Latency overhead < 5 ms p99.
10-100k RPS per node.
99.99% availability.

Capacity estimation#

50k RPS, 4 KB req → 200 MB/s.
JWT verify: ~10-50 µs (RS256 with cached JWKS).

API surface#

Public: gateway IS the API; OpenAPI spec defines routes.
Admin: POST /routes, POST /consumers, POST /plugins.

Data model#

Route{ id, hosts, paths, methods, service_id, plugins[] }
Service{ id, upstream, retry, timeout }
Consumer{ id, credentials[], plan_id }
Plan{ id, quota, rate_limit }

Trade-offs#

One big gateway vs BFF-per-client: one gateway is simpler ops; BFFs decouple clients but multiply infra.
Smart vs dumb gateway: too much logic at gateway = god component; keep business logic in services.
Sidecar mesh vs gateway: ingress gateway + service mesh (Istio/Linkerd) for east-west - overlap, decide boundary.

Real-world refs#

Kong, Tyk, Apigee, AWS API Gateway, Envoy + Istio, NGINX, Spring Cloud Gateway.

FAQ#

What is an API gateway and why use one?#

An API gateway is a single entry point that fronts internal microservices and handles TLS termination, authentication, rate limiting, routing, and aggregation, so clients see one consistent API.

What is the difference between an API gateway and a load balancer?#

A load balancer distributes traffic across replicas of the same service at L4 or L7. An API gateway adds protocol awareness like auth, rate limiting, request transformation, and per-route routing across many services.

Is an API gateway a single point of failure?#