Service Mesh#

Problem statement (interviewer prompt)

You have 200 microservices in 6 languages. Design a way to enforce mTLS, retries, circuit breakers and observability uniformly without modifying any application code. Compare sidecar mesh, per-host agent, and sidecarless (eBPF) approaches.

flowchart LR
  subgraph PodA[Pod A]
    A[Service A]
    SA([Sidecar proxy])
  end
  subgraph PodB[Pod B]
    SB([Sidecar proxy])
    B[Service B]
  end
  CP([Control plane])
  A --> SA -- mTLS, retries, traces --> SB --> B
  CP -.config.-> SA
  CP -.config.-> SB

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class A,SA,SB,B,CP service;

A service mesh adds a sidecar proxy next to every pod. Apps speak plain HTTP/gRPC to localhost; the sidecar handles mTLS, retries, traffic splitting, observability, and authz without any app changes.

flowchart TB
  subgraph DataPlane[Data plane - per pod sidecar]
    direction LR
    P1([Sidecar A<br/>Envoy / linkerd2-proxy])
    P2([Sidecar B])
    P3([Sidecar C])
    P1 --> P2
    P2 --> P3
  end

  subgraph ControlPlane[Control plane]
    XDS[xDS server<br/>route + cluster + listener config]
    CA[Cert authority<br/>SPIFFE / SPIRE]
    POLICY[Policy engine<br/>OPA / native]
    TELEM[Telemetry collector]
  end

  subgraph Features[Cross-cutting features]
    MTLS[mTLS everywhere]
    RETRY[Retries + timeouts + circuit breaker]
    TRAFFIC[Traffic split / canary / mirroring]
    AUTHZ[L7 authz]
    OBS[Distributed tracing + metrics]
  end

  ControlPlane -.config.-> DataPlane
  DataPlane --- Features

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class P1 edge;
    class P2,P3,XDS,CA,POLICY,MTLS,TRAFFIC,AUTHZ service;
    class RETRY datastore;
    class TELEM,OBS obs;

Why it exists#

Microservice resilience + security duplicates across every language. A mesh consolidates these concerns into a sidecar (or a per-host agent), uniformly enforced:

mTLS between every service, certs rotated automatically.
Retries with budget, timeouts, circuit breakers, outlier detection.
Traffic shifting for canary / blue-green / A-B.
Authz by service identity (SPIFFE ID), method, headers.
Telemetry: golden signals + traces, no app instrumentation needed.

Architecture choices#

	Sidecar (per pod)	Per-host agent	Sidecarless (eBPF)
Examples	Istio, Linkerd, Consul Connect	early Linkerd 1.x	Cilium Service Mesh
Resource cost	+1 container/pod	shared	kernel only
Mature	yes	aging	newer
Granularity	per app	per host	per socket

Istio data flow#

sequenceDiagram
  participant A as Service A
  participant SA as Envoy A
  participant SB as Envoy B
  participant B as Service B
  A->>SA: HTTP/gRPC localhost
  SA->>SB: mTLS, retries, tracing
  SB->>B: HTTP/gRPC localhost
  B-->>SB: response
  SB-->>SA: response
  SA-->>A: response

Ingress + mesh#

The mesh is for east-west traffic (service↔service). For north-south (internet ↔ service) you still need an ingress / API gateway. Many meshes ship an ingress gateway component that's just another Envoy.

When to use one#

30 services and growing.
Polyglot stack - Java, Go, Python, Node.
Zero-trust requirement (mTLS + identity-based authz).
You want canary / traffic shifting without app changes.

When to skip#

Monolith or a handful of services - library-based resilience (Resilience4j, Polly) is simpler.
Tight latency budget - the sidecar adds 0.5-2 ms per hop.

Common pitfalls#

mTLS misconfigurations silently drop traffic - start in permissive mode.
Resource overhead: 50-200 MB RAM per sidecar at scale.
Debug visibility: an extra network hop changes how curl localhost behaves.
Upgrades: control plane and proxies must move together; canary it.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Load balancer / GSLB	L4/L7 traffic distribution and failover	load-balancer
`HLD`	API gateway / BFF	single ingress, auth, rate limit, routing	api-gateway
`HLD`	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
`HLD`	Resilience patterns	timeout, retry, breaker, bulkhead, backpressure	resilience-patterns
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`HLD`	Service mesh	sidecar mesh, mTLS, traffic policy	service-mesh
`LLD`	Structural patterns	Adapter, Decorator, Facade, Proxy, Composite	structural-patterns

Quick reference#

Build vs buy#

Istio - most features, biggest learning curve.
Linkerd - simpler, Rust proxy, lower overhead.
Consul Connect - ties into HashiCorp stack.
AWS App Mesh / GCP Anthos / Azure ASM - managed flavours.
Cilium Service Mesh - eBPF, sidecarless, newer.

Identity model#

SPIFFE / SPIRE issues short-lived X.509 certs or JWT-SVIDs per workload.
Cert format spiffe://td/ns/<ns>/sa/<serviceaccount> makes "is service A allowed to call B?" a string match.

What stays in the app#

Business logic, validation, domain errors.
Retry semantics for non-idempotent business operations (mesh-level retry is for transport faults only).
Authn of the end-user (mesh authz is for service identity).

Refs#

Istio docs + "Istio in Action".
Linkerd 2 architecture posts.
SPIFFE specification.
"The Sidecar Pattern" - Bilgin Ibryam.

FAQ#

What does a service mesh actually do?#

It puts a transparent proxy next to every service so traffic policies, mTLS, retries, and observability are enforced uniformly across the fleet without changing application code.

Service mesh vs API gateway, what is the difference?#

An API gateway handles north-south traffic from external clients. A service mesh handles east-west traffic between internal services, focusing on identity, reliability, and telemetry.

When should I avoid a service mesh?#

Skip it for small fleets where the operational cost outweighs the benefit, single-language stacks where libraries can solve the same problems, or environments where sidecar overhead is unacceptable.

What is mTLS in a service mesh?#

Mutual TLS where each service identity is bound to a cert issued by the mesh control plane. Every call is encrypted and the peer identity is cryptographically verified.

How does a sidecarless mesh work?#

Instead of a per-pod proxy, eBPF or per-node agents intercept traffic in the kernel. It reduces resource cost but limits some advanced L7 features the sidecar model handles.

API Gateway: API gateways handle north-south traffic while service meshes manage east-west service communication
Observability: service meshes automatically emit distributed traces, metrics, and logs that feed observability platforms
Resilience Patterns: service meshes implement circuit breaking, retries, and timeouts as infrastructure-level resilience