Skip to content

Fine-Tune vs RAG#

The fine-tune vs RAG decision is the most-asked question in applied LLM work. Pick wrong and you spend $50K training a model that goes stale in two weeks; pick right and a $200 vector index outperforms a custom model.

flowchart TD
  Q[New capability needed] --> F{Stable knowledge?}
  F -- no, changes weekly --> R[RAG]
  F -- yes, frozen domain --> S{Style or skill?}
  S -- factual --> R
  S -- style, format, tone --> FT[Fine-tune]
  R --> H{Style also off?}
  H -- yes --> C[RAG + Fine-tune]
  H -- no --> R
  Q --> P{Quick fix?}
  P -- yes --> PE[Prompt engineer first]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class Q,P client;
    class F,S,H service;
    class R,FT,C,PE datastore;

Prompt engineering comes first. It is free, deploys in minutes, and a well-written system prompt + 3 few-shot examples often hits 80 percent of the target quality.

RAG is the answer when knowledge is fresh, large, or per-tenant. The model stays frozen; the index changes daily. You pay per query (embedding + retrieval + bigger prompt) but you can audit citations.

Fine-tuning is the answer when you need a new behavior, format, or compressed knowledge. Examples: classify support tickets into 200 internal categories, write legal contracts in your firm's style, or shrink a 70B prompt to a 7B model that learned the pattern.

The combination wins more often than either alone: RAG provides facts, the fine-tune handles format and tone. A medical assistant might fine-tune on doctor-style explanations and retrieve current drug interactions from a live database.

Quick rule of thumb: if the answer changes weekly, use RAG. If the format never changes, fine-tune. If neither, prompt-engineer.

Problem statement

Your product needs an LLM to answer questions about a 200-page employee handbook that is updated monthly, in your CEO's writing style, with citations. You have 5K example Q&A pairs, $20K of GPU budget, and 6 weeks. Should you fine-tune, build RAG, prompt-engineer, or do all three? What does the math say?

The fine-tune vs RAG decision comes down to five axes: freshness of knowledge, control over format, total cost of ownership, latency budget, and dataset size. This guide gives you the decision matrix, the cost math, and worked examples.

The decision matrix#

Axis Prompt eng RAG Fine-tune RAG + FT
Setup cost $0 $1K-10K $5K-100K $6K-110K
Per-query cost Lowest Medium (bigger prompt) Low (smaller prompt) Medium
Freshness Manual edit Re-embed docs Re-train Re-embed + occasional retrain
Citation No Yes, native No Yes
Format control Weak Weak Strong Strong
Knowledge capacity Context window Unbounded (vector DB) Bounded by training data Both
Latency One LLM call Retrieval + LLM One LLM call Retrieval + LLM
Hallucination risk Higher Lower (grounded) Same as base Lower
Time to ship Hours Days Weeks Weeks
Min dataset 0 100s of docs 1K-10K examples Both

Cost math, walked through#

Assume a 70B model at $3 per million input tokens and $15 per million output tokens, 1M queries per month, 500 input tokens and 200 output tokens per query.

Prompt engineering baseline:

cost = 1M * (500 * 3 + 200 * 15) / 1M = $1500 + $3000 = $4500/month

RAG: average prompt grows by 1500 tokens of retrieved context, plus $0.10 per 1M embedding tokens for the query (negligible) and vector DB hosting (~$500/month for 10M chunks).

cost = 1M * (2000 * 3 + 200 * 15) / 1M + 500 = $6000 + $3000 + 500 = $9500/month

Fine-tune to 7B: training cost = roughly $5K once, per-query cost on a self-hosted 7B = $0.30 per 1M input + $0.60 per 1M output (rough). Prompt shrinks back to 500 tokens because the model learned the pattern.

cost = 1M * (500 * 0.30 + 200 * 0.60) / 1M = $150 + $120 = $270/month
amortized = $270 + $5000 / 12 = $687/month

RAG + 7B fine-tune: combines retrieval cost with cheap inference.

cost = 1M * (2000 * 0.30 + 200 * 0.60) / 1M + 500 + $5000/12 = $600 + $120 + 500 + 417 = $1637/month

Crossover analysis: fine-tuning to a smaller model pays back the training cost in roughly 2 months at this scale. Below 100K queries per month, prompt engineering or RAG on a big API model wins. Above 1M queries, fine-tune to a small open model.

When to combine#

flowchart LR
  U([User query]) --> R[Retriever]
  R --> CTX[Top-k chunks]
  CTX --> FT[Fine-tuned model<br/>learned format + tone]
  U --> FT
  FT --> A([Answer with citations])

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class U,A client;
    class R,FT service;
    class CTX datastore;

Combine RAG with fine-tuning when:

  • The knowledge is fresh and the output format is specific (e.g. structured JSON in your schema)
  • You retrieved relevant text but the base model still hallucinates the format
  • You want to shrink the prompt: a fine-tuned model needs less in-context guidance, so the retrieved context dominates the input

Bryan Bischof's "RAG vs fine-tuning" guidance from the Hex team: start with RAG, fine-tune only when retrieval is good but generation is bad. If the retrieved chunks contain the right answer but the model still gets it wrong, you have a fine-tune problem. If the model is fine when handed the answer directly, your bug is retrieval.

Closed-form examples by use case#

Use case Best choice Why
Internal docs Q&A RAG Docs change, need citations
Sales email drafter Fine-tune Style and tone, no fresh facts needed
Code completion in proprietary framework RAG + FT Framework patterns (FT) + recent API docs (RAG)
Medical chatbot RAG Liability, must cite, freshness, audit trail
SQL generation for one schema Fine-tune Schema is stable, model learns it once
Customer support over knowledge base RAG KB updated daily, need citations
Legal contract drafting Fine-tune Firm style, vetted templates
News summarization Prompt eng Generic task, base model excels
Multi-tenant per-customer style Per-tenant LoRA + shared RAG Each customer's docs (RAG) + their style (LoRA)

Freshness vs control tradeoff#

quadrantChart
    title When to use what
    x-axis Low freshness need --> High freshness need
    y-axis Low control need --> High control need
    quadrant-1 RAG + Fine-tune
    quadrant-2 Fine-tune
    quadrant-3 Prompt engineering
    quadrant-4 RAG
    Sales email drafter: [0.2, 0.8]
    News summary: [0.7, 0.2]
    Internal docs QA: [0.85, 0.3]
    SQL gen: [0.2, 0.7]
    Medical chat: [0.7, 0.9]
    Legal drafting: [0.3, 0.85]

What fine-tuning cannot fix#

A common trap: teams try to fine-tune in fresh knowledge. It works for stable facts that recur often in training data, but it is the wrong tool for:

  • Time-sensitive facts (today's stock price)
  • Long-tail facts (one customer's specific contract terms)
  • Per-user data (your private email)
  • Auditable answers (regulators want citations, not weights)

For all of these, RAG is the answer. Fine-tuning compresses patterns, not facts.

What RAG cannot fix#

RAG cannot give the model new capabilities. If your base model cannot write valid SQL for any schema, retrieving the schema does not help much. RAG also cannot enforce a strict output format reliably; the model will drift toward its pretraining distribution.

For these, fine-tune.

Decision algorithm in code#

def choose_approach(use_case):
    if use_case.knowledge_freshness == "minutes" or use_case.knowledge_freshness == "days":
        base = "rag"
    elif use_case.knowledge_size_tokens > 100_000:
        base = "rag"
    else:
        base = "prompt"

    needs_format = use_case.format_strictness == "high"
    needs_style = use_case.style_consistency == "high"
    has_data = use_case.labeled_examples >= 1000
    high_volume = use_case.monthly_queries >= 500_000

    if (needs_format or needs_style or high_volume) and has_data:
        return f"{base} + fine-tune"
    return base

Production sequencing#

The pragmatic order: prompt engineer first, then RAG, then fine-tune. Each step exposes what the next step needs.

  1. Ship a prompt-engineered version in week 1. Measure where it fails.
  2. Add RAG if failures are knowledge-gap shaped (week 2-4).
  3. Collect 1000 high-quality examples from production. Fine-tune only if retrieved-context generations are still wrong.

Anthropic, OpenAI, and Cohere all publish guidance that mirrors this sequence. Anthropic's prompt caching can defer fine-tuning further by making long system prompts cheap.

Quick reference#

One-line rules#

  • Fresh knowledge -> RAG
  • Format / style -> Fine-tune
  • Neither -> Prompt engineer
  • Both -> Combine

Decision matrix#

Axis Prompt RAG Fine-tune RAG + FT
Setup cost $0 $1-10K $5-100K $6-110K
Per-query cost Low Medium Lowest Medium
Freshness Manual Re-embed Re-train Re-embed
Citation No Yes No Yes
Format control Weak Weak Strong Strong
Min data 0 100s docs 1K-10K examples Both
Time to ship Hours Days Weeks Weeks

Cost math at 1M queries/month#

  • Prompt only (70B API): $4500
  • RAG (70B API + retrieval): $9500
  • Fine-tuned 7B self-host: $687 amortized
  • RAG + FT 7B: $1637 amortized

Crossover: FT pays back in ~2 months at 1M qpm scale.

When fine-tune cannot help#

  • Time-sensitive facts
  • Long-tail facts
  • Per-user / per-tenant data
  • Audit / citation requirements -> Use RAG

When RAG cannot help#

  • Missing capability (SQL, code gen)
  • Strict output format / schema
  • Style and tone consistency -> Fine-tune

Sequencing in production#

  1. Prompt engineer week 1
  2. Add RAG if knowledge gaps (week 2-4)
  3. Fine-tune only if retrieved-context generations still wrong

Combine signals#

  • Retrieved text contains answer but model gets it wrong -> fine-tune problem
  • Model fine when handed answer directly -> retrieval problem
  • Both broken -> RAG + FT

Use case shortlist#

Use case Choice
Internal docs QA RAG
Sales email style FT
Medical chat RAG (liability)
SQL gen one schema FT
Multi-tenant style Per-tenant LoRA + RAG
News summary Prompt
Legal drafting FT

Refs#

  • Lewis et al., Retrieval-Augmented Generation (2020)
  • Anthropic, OpenAI, Cohere fine-tuning vs retrieval guidance
  • Hex team / Bryan Bischof, RAG vs Fine-tuning blog
  • Anthropic prompt caching docs

FAQ#

Should I fine-tune or use RAG?#

Use RAG when knowledge changes weekly or is too large to memorise. Fine-tune when you need a specific style, format, or skill that prompting cannot achieve. Many real systems combine both.

Is RAG cheaper than fine-tuning?#

Usually yes. A modest vector index plus a base model often costs hundreds of dollars to run, while a serious fine-tune run plus dedicated serving can cost tens of thousands and still go stale.

Can you combine RAG and fine-tuning?#

Yes. Fine-tune the base model to adopt your tone, terminology, and output schema, then use RAG to inject fresh facts at inference. This pattern is common in regulated and brand-sensitive products.

When does fine-tuning beat RAG?#

Fine-tuning wins for stable skills like JSON formatting, classification, summarisation style, or domain language patterns. It also wins on latency since you do not need a retrieval step.

Does fine-tuning teach an LLM new facts reliably?#

Not really. Fine-tuning shifts behaviour and style more than it injects facts, and it can cause hallucinations when probed on the edges of the training set. RAG is the safer way to ground answers.