Observability#

Problem statement (interviewer prompt)

Design the observability stack for a microservice deployment with 200 services. Cover metrics, structured logs, distributed traces, and continuous profiling; explain SLO/SLI/error budget; design alerts that page on user-visible breakage, not on internal CPU spikes.

flowchart LR
  APP[Service] --> M[(Metrics<br/>Prometheus)]
  APP --> L[(Logs<br/>Loki / ELK)]
  APP --> T[(Traces<br/>Jaeger / Tempo)]
  M --> G[Dashboards / Alerts]
  L --> G
  T --> G

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class APP service;
    class M,L,T datastore;
    class G obs;

Three pillars: Metrics (numbers over time), Logs (events), Traces (causal chains).

flowchart TB
  subgraph App[Application]
    INST([OpenTelemetry SDK<br/>auto + manual instrumentation])
    SLO[SLI / SLO definitions]
    EXEM[Exemplar trace ids on metrics]
  end

  subgraph Pipelines[Collection]
    OTEL[OTel Collector<br/>receivers, processors, exporters]
    AGENT[Per-host agent<br/>FluentBit / Vector / Promtail]
    SCRAPE[Prometheus scrape]
  end

  subgraph Metrics[Metrics tier]
    PROM[Prometheus / Thanos / Mimir / VictoriaMetrics]
    DD[Datadog / NewRelic]
    REC[Recording rules / aggregations]
    ALERT[Alertmanager]
  end

  subgraph Logs[Logs tier]
    LOKI[Loki / Elasticsearch / OpenSearch / Splunk]
    PARSE[Structured parsing<br/>JSON]
    REDACT[PII redaction]
    INDEX[Indexing strategy: labels + content]
    ARCH[Cold archive S3]
  end

  subgraph Traces[Traces tier]
    JAEG[Jaeger / Tempo / Honeycomb]
    SAMPL[Sampling head + tail]
    SPAN[Spans, links, baggage]
    PROP[W3C traceparent propagation]
  end

  subgraph Profiles[Continuous Profiling]
    PROF[Pyroscope / Parca / Pixie]
    CPU[CPU / heap / lock / off-cpu]
  end

  subgraph SLO_Stack[SLO & error budget]
    BURN[Burn rate alerts]
    MWMW[Multi-window multi-burn]
    OBJ[Targets: 99.9% etc]
  end

  subgraph UX[Dashboards & UX]
    DASH[Grafana / Kibana]
    NOTI[PagerDuty / Opsgenie]
    INCID[Incident commander / runbook]
  end

  INST --> OTEL
  INST --> SCRAPE
  AGENT --> OTEL
  OTEL --> Metrics
  OTEL --> Logs
  OTEL --> Traces
  SCRAPE --> PROM
  PROM --> REC --> ALERT
  ALERT --> NOTI
  Metrics --> DASH
  Logs --> DASH
  Traces --> DASH
  Profiles --> DASH
  EXEM -. link metric -> trace .-> Traces
  SLO --> BURN --> ALERT

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class AGENT,REC,PARSE,REDACT,INDEX,SAMPL,SPAN,PROF,CPU,MWMW,OBJ,INCID service;
    class ARCH storage;
    class INST,SLO,EXEM,OTEL,SCRAPE,PROM,DD,ALERT,LOKI,JAEG,PROP,BURN,DASH,NOTI obs;

SLI / SLO basics#

SLI = signal (e.g., "fraction of requests < 300 ms").
SLO = target (e.g., 99.9% over 28 days).
Error budget = 1 - SLO. Spend it on shipping.
Burn-rate alerts: page on fast burn (1 hr / 5%), warn on slow burn (6 hr / 10%).

Metric types (Prometheus model)#

Counter - monotonic, use rate() for per-second.
Gauge - value at a moment.
Histogram - bucketed; allows histogram_quantile.
Summary - pre-computed quantiles, not aggregatable.

Sampling#

Head sampling: decide at root span (random N%).
Tail sampling: decide after full trace (keep all errors, slow).
Adaptive sampling: keep enough per-route signal.

Logging discipline#

Structured JSON, severity, request id, user id (hashed), trace id.
Sample noisy lines; reserve INFO for state changes, DEBUG for diag only.
Don't log PII or secrets; redact at agent.

Pitfalls#

Cardinality explosion in Prometheus - beware unbounded labels (user id).
Logs as primary metric source - slow and expensive.
Alerts on symptoms not causes (user impact > CPU%).
"Alert fatigue" - page only on user-visible breakage.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Observability	metrics, logs, traces, SLOs	observability
`LLD`	Testing strategy	pyramid, doubles, TDD, contracts	testing-strategy
`LLD`	Behavioural patterns	Strategy, Observer, State, Command, Chain	behavioral-patterns

Quick reference#

Why three pillars (and what's missing)#

Metrics: cheap, aggregable, weak detail.
Logs: rich detail, costly, hard to query for trends.
Traces: causal chains across services.
4th: Continuous profiling (Pyroscope/Parca) - code-level resource attribution.
5th, emerging: eBPF events (network, syscalls) for kernel-side observability.

Cost rule of thumb#

Logs are by far the most expensive.
Sample logs and traces aggressively in steady state.
Always keep error traces / error logs.

Cardinality budget (Prometheus / Mimir)#

Labels multiply: user × route × method × status × dc.
Drop user-level labels; aggregate them server-side if needed.

What to instrument by default#

RED: Request rate, Errors, Duration (per route).
USE: Utilization, Saturation, Errors (per resource).
Golden Signals: latency, traffic, errors, saturation (SRE book).

Trace propagation#

W3C traceparent header in HTTP/gRPC.
Across queues, propagate via headers (Kafka, SQS message attributes).
Server logs include trace_id for join with traces.

Practical wins#

Exemplars on histogram metrics let you click p99 latency directly to a slow trace.
"Tail sampling" Collector keeps every error trace without overload.
Service map auto-generated from traces gives free architecture visibility.

Refs#

Google SRE Book + SRE Workbook (SLO/SLI chapters).
OpenTelemetry docs (https://opentelemetry.io).
"Distributed Systems Observability" - Cindy Sridharan.
Honeycomb blog series on tail sampling.
Prometheus / Grafana / Loki / Tempo docs.

FAQ#

What are the three pillars of observability?#

Metrics are numbers over time. Logs are discrete events with context. Traces are causal chains across services. Each answers a different question during incidents.

What is the difference between SLO, SLI, and error budget?#

An SLI is a measured metric like p99 latency. An SLO is the target like 99.9 percent. The error budget is the allowed failures, which gates new feature rollouts.

What is distributed tracing?#

Tracing tags each request with a trace ID and span IDs as it crosses services, so you can see the full causal chain, where time was spent, and which span errored.

Should you alert on CPU or on user-visible symptoms?#

Alert on user-visible symptoms like rising errors or saturated SLOs. CPU and queue depth belong in dashboards, not pagers, because they often spike without harming users.

What is OpenTelemetry?#

OpenTelemetry is a vendor-neutral SDK and protocol for metrics, logs, and traces. It lets you instrument once and export to Prometheus, Jaeger, Datadog, or any compatible backend.

Resilience Patterns: observability data drives circuit breakers, health checks, and adaptive resilience patterns
Service Mesh: service meshes emit the distributed traces, metrics, and logs that observability platforms ingest
API Gateway: API gateways are a primary source of request/response telemetry for observability pipelines