Skip to content

LLM Evals#

LLM evaluation evals are the test suite for AI systems: a repeatable measurement of quality that lets you ship model swaps, prompt changes, and new tools without crossing your fingers. Without evals, every change is a vibes-based migration. With evals, an LLM app becomes a normal software product.

flowchart LR
  Code[Prompt / model change] --> Run[Eval runner]
  Set[(Golden dataset<br/>versioned)] --> Run
  Run --> Judge[LLM-as-judge<br/>+ deterministic checks]
  Judge --> Score[Aggregate scores]
  Score --> Gate{Pass<br/>threshold}
  Gate -->|yes| Deploy[Deploy]
  Gate -->|no| Block[Block PR]

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
  class Code client;
  class Set datastore;
  class Run,Judge compute;
  class Score,Gate service;
  class Block,Deploy queue;

The core ingredients:

  • Golden dataset: 50-2000 input examples with expected behavior. Curated by humans, versioned in git or a dataset store.
  • Eval functions: a mix of cheap deterministic checks (regex, JSON validation, exact match) and LLM-as-judge prompts for fuzzy quality (factuality, helpfulness, tone).
  • Frameworks: ragas for RAG, Langfuse and Arize Phoenix for tracing and online evals, promptfoo and DeepEval for local CI, Braintrust for hosted.
  • CI integration: the eval runs on every PR. A regression past threshold blocks the merge.

Evals are how you turn an LLM feature into a sustainable engineering project instead of a one-shot demo.

Problem statement

A team has a RAG-powered support agent in production. Every time someone tweaks the prompt or swaps the embedding model, regressions land silently and surface as user complaints days later. Design the eval system that catches quality drops at PR time, runs in CI, and gives the team confidence to ship daily.

LLM evaluation evals are the answer: a repeatable, versioned, scored measurement of LLM behavior that gates deploys the same way unit tests gate code.

Why evals are unusual#

Unlike unit tests, LLM evals have:

  • Fuzzy correctness: there is no single right answer for "is this summary good?"
  • Stochastic outputs: the same input gives different outputs across runs.
  • Cost per test: every eval call costs tokens; you cannot run 100k tests per PR.
  • Distribution shift: real user queries evolve faster than test sets.

The standard answer is a tiered system: cheap deterministic checks first, expensive LLM-as-judge second, periodic human review third.

Golden test sets#

A golden set is a curated (input, expected_behavior) collection. Build it from three sources:

  1. Hand-written cases: the 50-100 hardest examples you can think of, including edge cases and known failure modes.
  2. Production traces: sampled real user interactions, especially flagged ones. Use Langfuse or Phoenix to capture.
  3. Synthetic generation: an LLM generates variants of seed examples for breadth, then a human reviews a slice.
# Illustrative dataset row
{
  "id": "refund-001",
  "input": "I want my money back on order ORD-123456",
  "expected_action": "create_refund",
  "expected_fields": {"order_id": "ORD-123456"},
  "tags": ["refund", "happy_path"],
  "version": "2026-04-15"
}

Version the set in git (small) or a dataset registry like Hugging Face Datasets, Argilla, or Braintrust (large). Tag each row so you can slice quality by intent, language, customer tier, etc.

Eval functions#

Deterministic checks (cheap)#

  • Exact match for classification tasks.
  • JSON schema validation on structured outputs (see Guardrails).
  • Regex / substring for required disclaimers, blocked phrases, citation IDs.
  • Latency, token cost, retry count as numerical SLOs.
  • Embedding similarity to a reference answer (cosine over text-embedding-3-large).

LLM-as-judge (expensive, fuzzy)#

Use a stronger or peer model to score outputs along rubrics. A typical judge prompt asks for a 1-5 score plus reasoning.

JUDGE_PROMPT = """Rate the assistant's reply on a scale of 1-5 for FACTUALITY
against the ground-truth answer. Output JSON: {"score": int, "reason": str}.

User question: {question}
Ground truth: {truth}
Assistant reply: {reply}"""

Judges drift, so always pin the judge model version and re-validate the judge against a small human-labeled set every quarter.

Bias mitigation for LLM-as-judge#

LLM judges have well-known biases (Zheng et al., MT-Bench paper):

  • Position bias in pairwise comparison: the first option wins more often. Mitigate by averaging both orderings.
  • Verbosity bias: longer answers score higher. Add a length-normalization rubric or use a stronger judge.
  • Self-preference: a model judges its own outputs higher. Use a different model family as judge.
  • Style over substance: well-formatted but wrong answers can win. Pair the judge with deterministic factuality checks.

ragas for RAG#

ragas provides off-the-shelf metrics tailored to RAG:

Metric What it measures
faithfulness Does the answer stay grounded in retrieved context?
answer_relevancy Does the answer address the question?
context_precision Are retrieved chunks ranked by relevance?
context_recall Did retrieval pull all needed chunks?
answer_correctness Does the answer match the gold reference?

Run on a held-out RAG eval set; track each metric independently so you can tell whether the regression is in retrieval, ranking, or generation.

Regression eval suite in CI#

sequenceDiagram
  participant Dev
  participant PR as GitHub PR
  participant CI
  participant Eval as Eval runner
  participant Track as Langfuse / Phoenix

  Dev->>PR: change prompt or model
  PR->>CI: trigger pipeline
  CI->>Eval: run golden set
  Eval->>Track: log per-row scores
  Track-->>Eval: aggregate metrics
  Eval-->>CI: pass / fail vs baseline
  alt pass
    CI-->>PR: green check
  else regression
    CI-->>PR: block, link to failing rows
  end

Run the smallest evals on every PR (50-100 examples, 1-3 minutes). Run the full suite nightly (1000+ examples). Keep a canary subset that runs continuously against production.

Dataset versioning and the eval workflow#

Treat eval sets like schemas: they grow, but old rows stay reproducible. A workable layout:

evals/
  datasets/
    refunds_v3.jsonl
    refunds_v3.lock      # checksum
  prompts/
    agent_v12.txt
  metrics/
    accuracy.py
    faithfulness.py
  ci/
    smoke.yml            # 100 rows on every PR
    full.yml             # 2000 rows nightly

Workflow:

  1. Engineer changes prompt or model in PR.
  2. CI runs smoke eval, posts diff vs main on PR.
  3. Reviewer inspects the per-row diff (where did scores drop?).
  4. If green, merge and trigger full eval.
  5. Production traces feed new failure cases back into the dataset.

This is eval-driven development: the dataset is the spec, every change has measurable behavior delta.

Cost of evals#

Evals are not free. A rough budget:

Item Approx cost per run
100-row smoke (Sonnet target + judge) $0.50 - $2
2000-row nightly $10 - $40
ragas full pass $5 - $20
Human review (1 reviewer-hour per 100 rows) engineer time

Strategies to control cost:

  • Cache target-model outputs when the inputs and prompts have not changed.
  • Use cheaper models as judges where reliable (Haiku, gpt-4o-mini).
  • Sample large datasets per run; only run the full set on release branches.
  • Use deterministic checks first; only invoke LLM judge for fuzzy axes.

Tooling map#

Tool Use
ragas RAG-specific metrics
Langfuse Trace + online eval + dataset store, open source
Arize Phoenix Trace + offline eval, open source
promptfoo YAML-driven local eval, CI-friendly
DeepEval pytest-style LLM unit tests
Braintrust Hosted dataset + eval + scorecard
OpenAI Evals Reference framework, model-graded templates

Production failure modes#

  • Eval set rot. A 6-month-old dataset no longer reflects production traffic. Continuously sample prod traces back into the set.
  • Judge model drift. Provider silently updates the judge model; scores shift. Pin to dated snapshot versions.
  • Leak between train and eval. Synthetic data generated by the same model under test contaminates results. Track provenance.
  • Single-number tyranny. A 92% overall score hides a 50% regression on a critical intent. Always slice metrics by tag.
  • Overfitting the prompt to the eval. Engineers tune prompts until eval is green but real users complain. Hold out a sealed prod-traffic set.

Quick reference#

TL;DR#

Golden dataset + scored eval functions + CI gate. The eval set is the spec.

Dataset sources#

  • Hand-written hard cases (50-100 to start).
  • Sampled production traces (Langfuse / Phoenix).
  • Synthetic variants with human-reviewed slice.

Eval function tiers#

Tier Examples Cost
Deterministic Schema valid, regex, exact match, latency Cheap
Embedding sim Cosine to reference Low
LLM-as-judge 1-5 rubrics for factuality, tone Expensive
Human review Sampled rows per release Engineer time

RAG metrics (ragas)#

  • faithfulness, answer_relevancy
  • context_precision, context_recall
  • answer_correctness

Judge biases to mitigate#

  • Position bias: average both orderings.
  • Verbosity bias: length normalize.
  • Self-preference: different model family.
  • Style over substance: pair with deterministic checks.

CI pattern#

  • Smoke (100 rows) on every PR, ~2 min.
  • Nightly full (1000-2000 rows).
  • Canary subset runs against prod continuously.
  • Block PR on regression vs main baseline.

Versioning#

  • Datasets in git (small) or Argilla / HF / Braintrust.
  • Tag rows by intent, language, tier.
  • Pin judge model snapshot, re-validate quarterly.

Cost control#

  • Cache target outputs across unchanged prompts.
  • Cheaper judges (Haiku, mini) where reliable.
  • Sample large sets; full only on release.
  • Deterministic first, LLM judge last.

Tools#

  • ragas, Langfuse, Phoenix, promptfoo, DeepEval, Braintrust, OpenAI Evals.

Watch-outs#

  • Eval set rot, continuously refresh from prod.
  • Aggregate scores hide intent regressions, slice by tag.
  • Overfitting prompts to eval, keep a sealed holdout.
  • Synthetic-data contamination, track provenance.

Refs#

  • ragas paper and docs (Es et al., 2023)
  • Zheng et al. - LLM-as-Judge with MT-Bench (2023)
  • Langfuse, Arize Phoenix open-source docs
  • OpenAI Evals repository
  • Hamel Husain - "Your AI product needs evals"

FAQ#

What are LLM evals?#

LLM evals are repeatable, scored tests that measure quality on a versioned golden dataset. They combine deterministic checks like regex and JSON validity with LLM-as-judge scoring for fuzzier criteria.

How does LLM-as-judge work?#

A stronger model scores another model's output against a rubric, often comparing two candidates pairwise. Calibrate the judge with human-labelled examples and check inter-rater agreement before trusting it.

What is ragas used for?#

Ragas is an open-source framework for evaluating RAG systems on metrics like faithfulness, answer relevancy, context precision, and context recall. It produces scores you can gate pull requests on.

How do you stop LLM regressions in production?#

Run evals on every prompt or model change in CI. Track scores per dataset version, alert on threshold drops, and block merges that fail. Add new failure cases to the golden set after every incident.

How big should an LLM eval set be?#

Start with 50 to 200 hand-crafted examples that cover the dominant intents and known edge cases. Grow from there using real production logs labelled by domain experts. Quality beats quantity.