LLM Tracing and Observability#

LLM tracing observability turns an opaque agent into a debuggable tree. A user request becomes one root span; every nested LLM call, tool call, retrieval, and rerank becomes a child span. You see latency, cost, prompts, and responses end to end.

flowchart TB
  R[Root: chat_request<br/>2.3s, $0.04] --> A[Agent.run<br/>2.2s]
  A --> L1[LLM.chat call 1<br/>800ms, $0.01]
  A --> T1[Tool: search_docs<br/>120ms]
  A --> L2[LLM.chat call 2<br/>900ms, $0.02]
  A --> T2[Tool: web_fetch<br/>200ms]
  A --> L3[LLM.chat final<br/>180ms, $0.01]
  T1 --> R1[Retriever.search<br/>80ms]
  T1 --> RR[Reranker.score<br/>40ms]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class R client;
    class A service;
    class L1,L2,L3 compute;
    class T1,T2,R1,RR service;

Each span carries attributes: llm.model, llm.input_tokens, llm.output_tokens, llm.cost_usd, tool.name, tool.error. OpenTelemetry has draft GenAI conventions; Langfuse, Arize, and LangSmith add LLM-specific fields on top.

Replay debugging is the killer feature. From a stored trace, fork at any LLM span, swap the prompt or model, and re-run. The first version of an agent often needs 10 to 20 replays to converge.

Eval-on-trace binds an automatic check (LLM-as-judge, regex, or function) to each span. The trace UI shows red where the check failed. Patterns: factuality on retrieval-augmented spans, format compliance on JSON spans, length and tone on final responses.

Dataset extraction closes the loop: traces from production become the next fine-tuning dataset. Filter to high-quality interactions (judged good, user thumbs-up, low-cost), export prompt + response pairs.

Problem statement

Your agent product handles 5,000 requests per hour. P95 latency drifted from 3s to 8s overnight. Users complain about wrong answers but you cannot reproduce. Existing APM tools show CPU and memory; they do not show prompts, token counts, tool calls, or model versions. Design an LLM tracing observability stack that diagnoses latency, cost, and quality regressions in minutes.

The trace tree#

flowchart TB
  R([HTTP /chat]) --> H[Handler.run]
  H --> AG[Agent.execute]
  AG --> P1[Planner LLM<br/>model: claude-haiku<br/>120ms, $0.001]
  AG --> RAG[Retriever.search<br/>180ms]
  RAG --> EM[Embed query<br/>40ms]
  RAG --> AN[ANN search<br/>50ms]
  RAG --> RR[Rerank<br/>90ms]
  AG --> L1[Synth LLM<br/>model: claude-opus<br/>1.4s, $0.02]
  AG --> T1[Tool: send_email<br/>250ms]
  AG --> L2[Reflect LLM<br/>320ms, $0.005]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class R client;
    class H,AG,RAG,T1 service;
    class P1,L1,L2,EM,AN,RR compute;

The trace is a tree of spans. Each span has a start time, end time, attributes, and events. The root span is the user-visible request; everything else is causally nested under it.

OpenTelemetry GenAI semantic conventions#

OpenTelemetry has draft gen_ai.* conventions. The minimum useful set:

Attribute	Example	Why
`gen_ai.system`	`anthropic`, `openai`	Vendor
`gen_ai.request.model`	`claude-opus-4-7`	Routed model id
`gen_ai.response.model`	`claude-opus-4-7-20260101`	Actual served version
`gen_ai.request.temperature`	`0.7`	Sampling param
`gen_ai.usage.input_tokens`	`2480`	Billed input
`gen_ai.usage.output_tokens`	`412`	Billed output
`gen_ai.usage.cost_usd`	`0.02`	Computed cost
`gen_ai.response.finish_reasons`	`["stop"]`	Termination
`gen_ai.prompt`	(event)	Raw prompt body
`gen_ai.completion`	(event)	Raw response body

Prompts and completions live as events on the span, not attributes. This matters because span attributes are indexed; events are typically not. Putting a 50KB prompt as an attribute breaks the index.

Instrumentation code#

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import anthropic

tracer = trace.get_tracer("my-agent")

def chat(messages):
    with tracer.start_as_current_span("anthropic.messages.create") as span:
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", "claude-opus-4-7")
        span.set_attribute("gen_ai.request.temperature", 0.7)
        span.add_event("gen_ai.content.prompt", {"messages": str(messages)})

        try:
            client = anthropic.Anthropic()
            resp = client.messages.create(
                model="claude-opus-4-7",
                max_tokens=1024,
                messages=messages,
            )
            span.set_attribute("gen_ai.response.model", resp.model)
            span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
            cost = (resp.usage.input_tokens * 3 + resp.usage.output_tokens * 15) / 1_000_000
            span.set_attribute("gen_ai.usage.cost_usd", cost)
            span.set_attribute("gen_ai.response.finish_reasons", [resp.stop_reason])
            span.add_event("gen_ai.content.completion", {"text": resp.content[0].text})
            return resp
        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise

Libraries that do this automatically: Langfuse (@observe() decorator), LangSmith (@traceable), OpenLLMetry (auto-instrumentation), Phoenix (OTel-native).

Sampling and storage#

flowchart LR
  T[10K traces/min] --> S{Sampler}
  S -- 100% errors --> A[(Hot store<br/>7 days)]
  S -- 100% slow --> A
  S -- 1% baseline --> A
  S -- 100% bad eval --> A
  A --> CO[Cold archive<br/>S3 + parquet<br/>1 year]
  A --> EX[Export to eval / FT dataset]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    class S service;
    class A datastore;
    class CO,EX storage;

LLM traces are big (prompts and completions are KB to MB each). Strategy:

Tail-based sampling: keep 100% of errors, 100% of P99 latency, 100% with failed eval, ~1% baseline
Hot store: 7 to 30 days, queryable (Langfuse, ClickHouse, Postgres)
Cold archive: Parquet on S3, accessed for FT data extraction

Cost reality: at 10K traces per minute, full retention is roughly 5 TB per day. Sampling brings this to ~50 GB per day, very tractable.

Eval-on-trace#

Bind a check to each span class. The check runs async, writes a score back, and surfaces in the trace UI.

from langfuse.decorators import observe

@observe(name="rag.synth")
def synthesize(question, chunks):
    answer = llm_call(prompt(question, chunks))
    # Bound eval runs in background worker
    langfuse.score(
        name="faithfulness",
        value=faithfulness_judge(answer, chunks),
    )
    return answer

def faithfulness_judge(answer, chunks):
    # LLM-as-judge: did the answer cite only what's in the chunks?
    judge_resp = haiku_call(f"Is this answer faithful to the sources?\n\nAnswer: {answer}\n\nSources: {chunks}")
    return parse_score(judge_resp)

Common eval categories:

Span type	Eval
Retrieval	Recall@k against gold, MRR
Generation	Faithfulness, helpfulness (LLM judge)
Structured output	JSON schema validity
Tool call	Argument schema validity, idempotency
End-to-end	Task success (LLM judge or human)

Replay debugging#

sequenceDiagram
  autonumber
  participant U as User
  participant T as Trace store
  participant D as Dev
  participant L as LLM
  U->>T: trace 7f3a (bad answer)
  D->>T: fork at synth span
  D->>L: same prompt, model=claude-opus
  L-->>D: response A
  D->>L: variant prompt, model=claude-opus
  L-->>D: response B
  D->>T: save A and B as variants
  D->>D: compare to ground truth

A trace store with replay lets you:

Open a failed trace
Pick any LLM span
Modify prompt / model / temperature / tools
Re-run that span and downstream
Diff against original

Langfuse Playground, LangSmith Playground, and Anthropic Workbench all do this. Indispensable for prompt engineering.

Dataset extraction from traces#

Production traces are the highest-quality fine-tuning data you can get. Pipeline:

def extract_sft_dataset(start, end, min_quality=0.8):
    traces = trace_store.query(
        start=start, end=end,
        filters={
            "user_feedback": "thumbs_up",
            "eval.helpfulness": ">= 0.8",
            "eval.faithfulness": ">= 0.9",
            "tokens": "< 4000",  # cost control
        },
    )
    return [
        {"messages": t.messages, "completion": t.final_response}
        for t in traces
    ]

Filters to apply:

User thumbs-up or explicit positive feedback
Auto-eval passed (faithfulness, format)
Non-degenerate (token count in a sane band)
Diverse (deduplicate by embedding similarity)

For DPO data: pair traces with the same query but different ratings (thumbs-up vs thumbs-down) as chosen vs rejected.

Cost dashboards from spans#

Aggregate gen_ai.usage.cost_usd by gen_ai.request.model, tenant_id, route, and time bucket. A typical chart that catches incidents fast: cost-per-request P50, P95 by model, day over day.

SELECT
  date_trunc('hour', timestamp) AS hour,
  attributes->>'gen_ai.request.model' AS model,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY (attributes->>'gen_ai.usage.cost_usd')::float) AS p50,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY (attributes->>'gen_ai.usage.cost_usd')::float) AS p95,
  COUNT(*) AS requests
FROM spans
WHERE name = 'anthropic.messages.create'
  AND timestamp > NOW() - INTERVAL '24 hours'
GROUP BY 1, 2
ORDER BY hour DESC;

Privacy and PII#

Prompts and completions often contain PII. Mitigations:

Strip PII before logging (regex or presidio)
Hash user IDs in attributes
Encrypt prompt/completion blobs at rest
Separate retention: short for raw, long for hashed
Honor per-tenant opt-out

Production failure modes#

Failure	Cause	Fix
Missing nested LLM spans	Async lost OTel context	Use `contextvars` or `with_current_context`
Spans without cost	Streamed call, usage not captured	Subscribe to stream end event, fill on completion
Trace explosion (too many spans)	Per-token spans, recursive agent	Sample, or cap children per span
PII in indexed attributes	Prompt set as attribute, not event	Move to events, scrub
Dropped spans	Exporter backpressure	Use OTel batch exporter, monitor queue

Tooling landscape (2026)#

Tool	Strength
Langfuse	Self-hostable, full LLM-native, free tier
LangSmith	Tight LangChain integration
Arize / Phoenix	OTel-native, strong eval
Helicone	Drop-in proxy, low effort
OpenLLMetry (Traceloop)	OTel wrappers across providers
Honeycomb / Datadog	General APM with custom GenAI dashboards

Quick reference#

Mental model#

Each request = root span. Each nested LLM, tool, retrieval, rerank = child span. Tree with attributes for model, tokens, cost.

OTel GenAI attributes#

gen_ai.system: anthropic, openai
gen_ai.request.model: routed id
gen_ai.response.model: served version
gen_ai.request.temperature
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cost_usd
gen_ai.response.finish_reasons
gen_ai.prompt / gen_ai.completion: as events, not attributes

Why events for prompt/completion#

Span attributes are indexed (small, structured). Events are not (big, blob-like). Indexing 50KB prompts breaks the index.

Sampling#

100% errors
100% P99 latency
100% failed eval
~1% baseline
Tail-based sampler in collector

Storage tiers#

Hot: 7-30 days, queryable (Langfuse, ClickHouse)
Cold: Parquet on S3, 1 year, for FT data extraction

Eval-on-trace#

Bind a check per span class: | Span | Eval | |---|---| | Retrieval | Recall@k, MRR | | Generation | Faithfulness, helpfulness (LLM judge) | | Structured out | JSON schema | | Tool call | Arg schema, idempotency | | End-to-end | Task success |

Replay debugging#

Open failed trace
Fork at any LLM span
Swap prompt, model, temp, tools
Re-run downstream
Diff vs original Langfuse / LangSmith / Anthropic Workbench all support.

Dataset extraction#

Filter traces by: - User thumbs up - Eval pass (faithfulness, format) - Non-degenerate token counts - Diverse (dedupe by embedding)

Output: SFT (messages, completion) or DPO (chosen vs rejected from same query).

Cost dashboard query#

Aggregate cost_usd by model, tenant, route, hour. Alert on P95 cost-per-request spike.

Privacy#

PII strip before log
Hash user IDs
Encrypt blobs at rest
Separate retention for raw vs hashed
Per-tenant opt-out

Common failure modes#

Lost context in async -> contextvars
No cost on streamed -> fill on stream end
Span explosion -> sample, cap children
PII as attribute -> move to event, scrub
Dropped spans -> batch exporter queue depth

Tools (2026)#

Langfuse: self-host, free tier, LLM-native
LangSmith: LangChain-tight
Arize / Phoenix: OTel + eval
Helicone: proxy, low effort
OpenLLMetry: OTel wrappers
Honeycomb / Datadog: general APM + custom

Refs#

OpenTelemetry GenAI semantic conventions (draft)
Langfuse, LangSmith, Phoenix docs
OpenLLMetry / Traceloop project

FAQ#

How do you trace an LLM application?#

Wrap each LLM call, tool call, and retrieval in a span with parent-child relationships. Record input, output, model id, tokens, and cost as span attributes, then ship to a tracing backend like Langfuse or Arize.

What is the difference between LLM tracing and logging?#

Logs are flat event streams. Traces are nested trees that show parent-child causality across LLM calls, tools, and retrievers in one user request, which is what you need to debug agent runs.

How do you track LLM cost per request?#

Record input and output tokens on every LLM span, multiply by the model price, and aggregate over the request's trace id. Most tracing tools include cost as a first-class attribute.

Should I use OpenTelemetry for LLM tracing?#

Yes when you already standardise on OTel for the rest of your services. The OpenTelemetry GenAI conventions define standard attributes for prompts, tokens, and tool calls so traces work in any backend.

Which LLM tracing tool should I pick?#

Langfuse and Arize Phoenix are popular open-source choices. LangSmith integrates tightly with LangChain. Pick based on whether you need self-hosting, eval features, or framework integrations.

Observability: general tracing principles
LLM Evaluation: the eval signals you bind to spans
Distributed Tracing: OpenTelemetry foundations

Video walkthrough

Langfuse Intro: Observability & Tracing Deep Dive : via Langfuse

LLM Tracing and Observability#

The trace tree#

OpenTelemetry GenAI semantic conventions#

Instrumentation code#

Sampling and storage#

Eval-on-trace#

Replay debugging#

Dataset extraction from traces#

Cost dashboards from spans#

Privacy and PII#

Production failure modes#

Tooling landscape (2026)#

Quick reference#

Mental model#

OTel GenAI attributes#

Why events for prompt/completion#

Sampling#

Storage tiers#

Eval-on-trace#

Replay debugging#

Dataset extraction#

Cost dashboard query#

Privacy#

Common failure modes#

Tools (2026)#

Refs#

FAQ#

How do you trace an LLM application?#

What is the difference between LLM tracing and logging?#

How do you track LLM cost per request?#

Should I use OpenTelemetry for LLM tracing?#

Which LLM tracing tool should I pick?#

Related Topics#

Video walkthrough