RAG Patterns

RAG patterns retrieval augmented generation ground an LLM's answer in external documents fetched at query time. The pattern family spans from one-shot vector lookup to multi-hop graph traversal; picking the right one depends on the corpus and the query shape.

flowchart LR
  Q([Query]) --> R[Retriever]
  R --> RR[Reranker]
  RR --> G[Generator LLM]
  D[(Doc index)] --> R
  G --> A([Grounded answer])

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class Q,A client;
    class R,RR,G service;
    class D datastore;

Naive RAG is one embedding lookup then prompt the LLM with top-K chunks. It works surprisingly well as a baseline but breaks on acronyms, multi-hop questions, and domain jargon. Every advanced pattern addresses one of those breakages.

Hybrid search combines dense embeddings with BM25 keyword scoring and fuses results with reciprocal rank fusion. HyDE has the LLM hallucinate an ideal answer first, then retrieves docs similar to that hypothetical. Query rewriting uses an LLM to expand or split the question. GraphRAG builds an entity graph from the corpus and traverses it for multi-hop reasoning. Late chunking embeds the full document, then slices the embedding into chunk representations to preserve cross-chunk context.

There is no single best pattern. A production RAG system usually layers two or three: hybrid retrieval + cross-encoder reranking is the workhorse baseline; HyDE or GraphRAG sits on top for harder queries.

Problem statement

Your LLM does not know your private docs and hallucinates when forced to guess. You need a retrieval layer that pulls the right passages from a million-document corpus, in under 300ms, and feeds them to the model with enough context to answer correctly. A single embedding-and-fetch is the obvious first try; it fails on acronyms, jargon, and multi-hop questions. Design a layered pipeline.

RAG patterns retrieval augmented generation are a family of designs that vary in two dimensions: how the query is transformed, and how the index is structured. We walk the most important six.

The pipeline shape

flowchart LR
  subgraph Indexing[Index pipeline, offline]
    D[Docs] --> Ch[Chunker] --> Emb[Embedder] --> I[(Vector + BM25 index)]
  end
  subgraph Query[Query pipeline, online]
    Q[User query] --> QT[Query transform]
    QT --> Ret[Retriever]
    Ret --> RR[Reranker]
    RR --> Ctx[Context builder]
    Ctx --> LLM[Generator LLM]
    LLM --> Ans[Answer]
  end
  I --> Ret

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class Q,Ans client;
    class QT,Ret,RR,Ctx,LLM service;
    class Ch,Emb compute;
    class I datastore;

Every pattern below changes one box in this diagram.

Naive RAG

def naive_rag(question: str) -> str:
    q_vec = embed(question)
    chunks = vector_db.search(q_vec, top_k=5)
    prompt = f"Answer using these passages:\n{format(chunks)}\n\nQ: {question}"
    return llm.complete(prompt)

Top-K cosine similarity, stuff into prompt, generate. Decent on FAQ-style queries against clean docs. Fails when:

• Query uses an acronym the embedder never saw ("SRP", "DLQ")
• Answer requires combining two chunks that do not share vocabulary
• Top-K includes high-similarity but off-topic chunks (the "lost in the middle" effect)

Hybrid search (dense + BM25)

flowchart LR
  Q[Query] --> D[Dense ANN]
  Q --> B[BM25]
  D --> R[RRF merge]
  B --> R
  R --> TopK[Top-K candidates]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class D,B,R service;

Run both retrievers, fuse with Reciprocal Rank Fusion: score(d) = sum(1 / (k + rank_i(d))), typical k=60. Dense catches paraphrases, BM25 catches exact-term queries (codes, acronyms, names). RRF needs no score normalization, which is why it dominates over weighted sums.

def hybrid_search(question: str, top_k: int = 20):
    dense_hits = vector_db.search(embed(question), top_k=50)
    bm25_hits = bm25_index.search(question, top_k=50)
    return rrf_merge(dense_hits, bm25_hits, k=60, top_k=top_k)

Hybrid is the production baseline. If you ship one pattern, ship this.

HyDE (Hypothetical Document Embeddings)

The asymmetry problem: queries are short, docs are long. Their embeddings live in different regions of the vector space. HyDE fixes this by generating a fake answer first and embedding that.

def hyde(question: str, top_k: int = 10):
    fake_answer = llm.complete(
        f"Write a plausible paragraph that answers: {question}"
    )
    return vector_db.search(embed(fake_answer), top_k=top_k)

Tradeoffs:

• Adds an LLM call to every query (latency + cost)
• Wins big on technical, jargon-heavy domains where the question wording barely matches the doc wording
• Loses on short factual queries where the hypothetical drifts

Query rewriting and decomposition

flowchart LR
  Q[Raw query] --> RW[LLM rewriter]
  RW --> Q1[Sub-query 1]
  RW --> Q2[Sub-query 2]
  RW --> Q3[Sub-query 3]
  Q1 --> R[Retriever]
  Q2 --> R
  Q3 --> R
  R --> Merge[Dedup and merge]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class RW,R,Merge service;

A short LLM call rewrites or splits the query before retrieval:

• Expansion: "DLQ in Kafka" becomes "dead letter queue in Apache Kafka" and "Kafka error handling"
• Decomposition: "Compare Postgres and MySQL replication" becomes two sub-queries

Works well when paired with hybrid retrieval. The cost is one extra LLM call (cheap with Haiku-class models).

GraphRAG

For multi-hop questions ("Which engineer reviewed the most PRs by the team that owns service X last quarter?"), embedding similarity alone is hopeless. GraphRAG builds an entity-relation graph from the corpus during indexing and traverses it at query time.

flowchart TB
  subgraph Build[Index time]
    D[Docs] --> EX[LLM entity extractor] --> G[(Knowledge graph)]
  end
  subgraph Query[Query time]
    Q[Query] --> EN[Entity linker] --> T[Graph traversal] --> P[Path passages]
    G --> T
  end

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class EX,EN,T service;
    class G datastore;

Microsoft's GraphRAG and Anthropic's "contextual retrieval" variations build community summaries over the graph, so global questions ("themes in this corpus") get answered too. Cost: indexing is expensive (LLM calls per chunk). Wins: dramatically better on synthesis and multi-hop questions.

Late chunking

Standard chunking embeds each chunk in isolation, losing the surrounding context. Late chunking embeds the full document with a long-context encoder, then slices the token embeddings into chunk-level vectors by mean-pooling token spans. Each chunk vector now carries information from the rest of the doc.

def late_chunking(doc: str, chunk_spans: list[tuple[int,int]]):
    token_embeddings = long_ctx_encoder(doc)  # shape: [n_tokens, d]
    chunks = []
    for (start, end) in chunk_spans:
        chunks.append(token_embeddings[start:end].mean(axis=0))
    return chunks

Best paired with semantic chunking. The cost is needing a long-context embedder (8K to 32K input); the win is much higher recall on cross-reference questions.

Reranking

Whatever the retriever, the top-50 candidates almost always need a second pass. A cross-encoder scores (query, passage) jointly and reorders:

def rerank(question: str, candidates: list[str], top_k: int = 5):
    scores = cross_encoder.predict([(question, c) for c in candidates])
    ranked = sorted(zip(scores, candidates), reverse=True)
    return [c for _, c in ranked[:top_k]]

Cross-encoders (Cohere Rerank, bge-reranker, jina-reranker) cost ~50ms for 50 pairs and lift top-5 precision by 10 to 20 points. Always rerank in production.

Choosing a pattern

Query shape	Recommended stack
Short FAQ over clean docs	Naive RAG + rerank
Mixed: jargon, acronyms, paraphrases	Hybrid + rerank
Heavy jargon, query-doc asymmetry	Hybrid + HyDE + rerank
Multi-part comparisons	Query decomposition + hybrid + rerank
Multi-hop, entity-driven	GraphRAG + hybrid for fallback
Long docs, cross-reference questions	Late chunking + hybrid + rerank

Production failure modes

• Retrieval recall ceiling: top-K is right but answer is in chunk K+1. Raise K, then rerank harder.
• Stale index: docs updated, embeddings not refreshed. Use change-data-capture into the indexer.
• Embedding model swap: re-encoding takes hours. Run dual indexes during cutover.
• Context overflow: 5 chunks at 1K tokens each is fine, 50 is not. Summarize before stuffing.
• Prompt injection via retrieved doc: a malicious page in the corpus rewrites instructions. Fence retrieved content in clear delimiters and warn the model.

Evaluation

Track three numbers:

1. Recall@K: did the ground-truth chunk make it into top-K?
2. MRR / nDCG: how high did it rank?
3. End-to-end answer quality: LLM-as-judge or human eval.

Recall@K is fastest to iterate on. End-to-end is what users see.

Quick reference

Mental model

Retrieve relevant chunks at query time, stuff into prompt, generate. The pattern is which retriever, which transform, which rerank.

Pattern menu

Pattern	When it wins
Naive RAG	Clean FAQ docs, baseline
Hybrid (dense + BM25)	Mixed queries, acronyms, names; production default
HyDE	Query-doc vocabulary asymmetry; jargon-heavy
Query rewriting / decomposition	Multi-part, ambiguous, expansion-needing queries
GraphRAG	Multi-hop, entity-driven, synthesis
Late chunking	Long docs, cross-reference questions

Fusion math (RRF)

score(d) = sum 1 / (k + rank_i(d)), k=60 typical. No score normalization needed.

Always rerank

Cross-encoder reranking (Cohere Rerank, bge-reranker) on top-50 lifts precision 10 to 20 points for ~50ms.

Pipeline stages

1. Chunk and embed (offline)
2. Build vector + BM25 indexes
3. Query transform (optional: HyDE, rewrite, decompose)
4. Retrieve top-50
5. Rerank to top-5
6. Build context, prompt LLM

Failure modes

• Recall ceiling (raise K, then rerank)
• Stale index (CDC into indexer)
• Embedding swap (dual-index cutover)
• Context overflow (summarize before stuffing)
• Prompt injection via retrieved doc (fence content, warn model)

Evaluation

• Recall@K: ground truth in top-K
• MRR / nDCG: ranking quality
• End-to-end: LLM-as-judge or human

Stack picker

• Default: Hybrid + cross-encoder rerank
• Add HyDE if query-doc vocab differs
• Add decomposition if questions are multi-part
• Switch to GraphRAG for entity-heavy multi-hop

Refs

• Gao et al., HyDE (2022)
• Microsoft, GraphRAG (2024)
• Jina, Late Chunking (2024)
• Cohere Rerank docs

FAQ

What is retrieval-augmented generation?

RAG retrieves relevant documents from an index at query time and includes them in the LLM prompt, so the answer is grounded in external context rather than only the model's training data.

What is the difference between naive RAG and hybrid RAG?

Naive RAG uses only vector search. Hybrid RAG combines BM25 keyword search with vector search and merges the results with reciprocal rank fusion, which improves recall on rare terms and proper nouns.

What is HyDE in RAG?

HyDE asks an LLM to write a hypothetical answer to the query, embeds that answer, and retrieves documents similar to it. This helps when the original query is too short to match good chunks.

What is GraphRAG?

GraphRAG builds a knowledge graph from the corpus and uses graph traversal plus vector search to answer multi-hop questions. It outperforms flat RAG on questions that span many documents.

Which RAG pattern should I use first?

Start with hybrid search plus a cross-encoder reranker. It is the strongest baseline for most corpora and lets you measure whether more advanced patterns like HyDE or GraphRAG are actually worth the complexity.

Related Topics

• RAG Chunking Strategies: the upstream input to every pattern here
• Vector Databases: the index layer
• Agent Loop and ReAct Pattern: RAG often runs as the first tool inside an agent loop