Skip to content

RAG Chunking Strategies#

RAG chunking strategies decide how a document is sliced into the units a retriever stores and returns. The chunker is the most underrated part of a RAG pipeline: a bad chunker caps retrieval quality no matter how strong the embedder or reranker is.

flowchart LR
  D[Long doc] --> CH{Chunker}
  CH -->|fixed| F[~512-token slices]
  CH -->|recursive| R[Split on \n\n, \n, '.', ' ']
  CH -->|semantic| S[Embedding boundary detection]
  CH -->|markdown| M[Heading-aware]
  CH -->|code| C[AST-aware]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class D client;
    class CH service;
    class F,R,S,M,C compute;

The two knobs that matter are chunk size and overlap. Larger chunks carry more context but are less precise (retrieval returns big blobs of mostly-irrelevant text). Smaller chunks are precise but lose context. Overlap (typically 10 to 20 percent) bridges chunk boundaries so a sentence split in half still appears intact in one chunk.

Fixed-size and recursive chunkers ignore content. Semantic chunkers cut on embedding-similarity drops between adjacent sentences. Late chunking flips the order: embed first, slice second, so chunk vectors carry whole-document context. Format-aware chunkers (markdown headings, code AST nodes) preserve structure that flat splitters destroy.

There is no universal best chunker, but there is a universal pattern: pick the chunker that respects the document's natural unit (a function, a section, a paragraph), then tune size and overlap on a held-out eval set.

Problem statement

You have 10,000 PDFs averaging 40 pages each. The embedder takes max 8K tokens. The retriever returns top-K chunks for the LLM. How you split documents determines whether retrieval ever surfaces the right answer. Design the chunking layer so that precision (retrieved chunks are relevant) and recall (the right chunk is in top-K) both stay high across heterogeneous content: prose, tables, code, markdown.

RAG chunking strategies sit upstream of every retrieval decision. The chunker's job is to produce units that are (a) self-contained enough to answer a question on their own, and (b) granular enough that top-K retrieval does not waste budget on irrelevant context.

The two-knob model#

flowchart LR
  subgraph Tradeoff
    direction LR
    Small[Small chunks<br/>high precision<br/>low context]
    Big[Big chunks<br/>low precision<br/>high context]
  end
  Small <--> Big

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class Small,Big service;
Knob Effect of increasing
Chunk size More context per hit, but each top-K result wastes more budget; recall up, precision down
Overlap Lower boundary loss, but storage cost up; duplicates in top-K

Typical defaults: 256 to 1024 tokens, 10 to 20 percent overlap.

Fixed-size chunking#

def fixed_size(text: str, size: int = 512, overlap: int = 64):
    tokens = tokenize(text)
    chunks = []
    for i in range(0, len(tokens), size - overlap):
        chunks.append(detokenize(tokens[i:i + size]))
    return chunks

Pros: trivial, parallelizable, predictable token budget. Cons: cuts mid-sentence, mid-table, mid-function. Always produces a baseline; rarely the best.

Recursive chunking#

LangChain's RecursiveCharacterTextSplitter made this canonical. Try separators in order: \n\n (paragraphs), \n (lines), . (sentences), then characters. Each call splits into pieces under the size limit; if a piece is still too big, recurse with the next separator.

def recursive(text: str, size: int = 512, separators=("\n\n", "\n", ". ", " ")):
    if len(text) <= size or not separators:
        return [text]
    sep, *rest = separators
    parts = text.split(sep)
    chunks, buf = [], ""
    for p in parts:
        if len(buf) + len(p) + len(sep) <= size:
            buf = (buf + sep + p) if buf else p
        else:
            if buf:
                chunks.append(buf)
            buf = p if len(p) <= size else None
            if buf is None:
                chunks.extend(recursive(p, size, rest))
                buf = ""
    if buf:
        chunks.append(buf)
    return chunks

Pros: respects natural boundaries opportunistically. Cons: still content-blind. A 400-token paragraph and a 400-token table are treated the same.

Semantic chunking#

Cut where adjacent sentences become dissimilar in embedding space.

flowchart LR
  S1[Sentence 1] --> E1[embed]
  S2[Sentence 2] --> E2[embed]
  S3[Sentence 3] --> E3[embed]
  S4[Sentence 4] --> E4[embed]
  E1 -.- |sim 0.91| E2
  E2 -.- |sim 0.88| E3
  E3 -.- |sim 0.42| E4
  E3 --> Cut[Cut here]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class S1,S2,S3,S4 service;
    class E1,E2,E3,E4 compute;
    class Cut service;
def semantic(text: str, threshold: float = 0.7):
    sents = split_sentences(text)
    embs = [embed(s) for s in sents]
    chunks, buf = [], [sents[0]]
    for i in range(1, len(sents)):
        if cosine(embs[i - 1], embs[i]) < threshold:
            chunks.append(" ".join(buf))
            buf = [sents[i]]
        else:
            buf.append(sents[i])
    if buf:
        chunks.append(" ".join(buf))
    return chunks

Pros: chunks are topically coherent. Cons: costs N embedding calls at index time; threshold is dataset-dependent.

Late chunking#

Standard order: chunk first, then embed each chunk in isolation. Each chunk's embedding is blind to the rest of the doc. Late chunking swaps the order:

flowchart LR
  D[Doc] --> LC[Long-context encoder] --> TE[Token embeddings] --> SL[Slice by chunk spans] --> CE[Chunk embeddings]

    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class LC,TE,SL,CE compute;
def late_chunk(doc: str, span_size: int = 256):
    token_embs = long_ctx_encoder(doc)  # [n_tokens, d], doc-aware contextual embeddings
    chunks = []
    for start in range(0, len(token_embs), span_size):
        span = token_embs[start:start + span_size]
        chunks.append(span.mean(axis=0))
    return chunks

The token embeddings already encode cross-chunk relationships (this is what self-attention does over the whole doc). Slicing afterward inherits that context. Requires a long-context embedder (Jina v3, BGE-M3, OpenAI text-embedding-3 with 8K+).

Markdown-aware chunking#

Markdown carries structure: H1, H2, H3, lists, code blocks. A naive splitter shatters these. A markdown-aware chunker keeps a heading breadcrumb with each chunk:

def markdown_chunk(md: str):
    chunks = []
    crumbs = []  # current heading stack
    for line in md.split("\n"):
        if line.startswith("#"):
            level = len(line) - len(line.lstrip("#"))
            crumbs = crumbs[:level - 1] + [line.strip("# ").strip()]
            continue
        if line.strip():
            chunks.append({"text": line, "headings": list(crumbs)})
    return merge_until_size(chunks, size=512)

Each chunk now carries its section path ("Architecture > Storage > Sharding"), which both helps retrieval (BM25 hits the heading text) and helps the LLM ground its answer.

Code-aware chunking#

Source code has its own natural units: functions, classes, methods. AST-based chunkers slice on those:

import ast

def python_chunk(source: str):
    tree = ast.parse(source)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            chunks.append({
                "name": node.name,
                "kind": type(node).__name__,
                "code": ast.get_source_segment(source, node),
                "docstring": ast.get_docstring(node) or "",
            })
    return chunks

For non-Python languages, tree-sitter gives a uniform AST API across 40+ languages. Code-aware chunking dramatically outperforms recursive chunking on code-search benchmarks because functions are the actual unit users ask about.

Overlap, sliding windows, and parent-child#

Three patterns for preserving cross-chunk context:

  1. Overlap: each chunk includes the last N tokens of the previous one. Cheapest. Default to ~15%.
  2. Sliding window: smaller stride than chunk size. Produces more chunks (storage cost) but never loses boundary content.
  3. Parent-child: small chunks for retrieval, large parent chunks for context. Retrieval hits a child; the LLM sees the parent.
flowchart TB
  D[Doc] --> P1[Parent 1<br/>1024 tokens]
  D --> P2[Parent 2<br/>1024 tokens]
  P1 --> C1[Child A<br/>256 tokens]
  P1 --> C2[Child B<br/>256 tokens]
  P2 --> C3[Child C<br/>256 tokens]
  C1 -. retrieved .-> A[Answer]
  P1 -. provided to LLM .-> A

    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class P1,P2 datastore;
    class C1,C2,C3 service;

Parent-child is the best of both worlds when storage is cheap. Retrieve on precise small chunks, serve big context to the LLM.

Picking a strategy#

Content type Recommended chunker
FAQ, support tickets Recursive, 256-512 tokens, 15% overlap
Long-form prose, books Semantic or late chunking
Markdown docs (this site) Markdown-aware with heading breadcrumb
Source code AST-based per function
Tables and structured data Per-row or per-record, not text chunking
Multi-modal (images + text) Caption images, chunk captions separately

Evaluation#

Build a small labeled set: question, ground-truth chunk. Measure Recall@K under different chunkers and sizes. A typical sweep:

for size in [128, 256, 512, 1024]:
    for overlap in [0, 32, 64, 128]:
        chunks = chunker(corpus, size=size, overlap=overlap)
        index = build_index(chunks)
        r10 = recall_at_k(index, eval_set, k=10)
        print(size, overlap, r10)

Stop when Recall@10 plateaus; bigger chunks beyond that just inflate cost.

Failure modes#

  • Table mid-cut: a 12-column table split into 2 chunks loses headers in chunk 2. Detect tables, keep intact.
  • Code mid-function: same problem. Always use AST for code.
  • Acronym at boundary: "...the DLQ" in chunk N, "...will retry" in chunk N+1. Overlap mitigates.
  • Tiny chunks dominate top-K: 50-token chunks are dense and over-rank. Set a minimum size.
  • Embedding drift after reindex: changing chunker invalidates the index. Dual-index during cutover.

Quick reference#

Mental model#

Chunker output is what the retriever can ever return. Bad chunks cap the pipeline regardless of embedder or reranker.

Two knobs#

  • Size: 256 to 1024 tokens typical. Bigger = more context, less precision.
  • Overlap: 10 to 20 percent. Mitigates boundary loss.

Chunker menu#

Strategy Idea Best for
Fixed-size Slice every N tokens Baseline only
Recursive Try \n\n, \n, ., in order Generic prose
Semantic Cut on embedding similarity drop Topically diverse docs
Late chunking Embed full doc, slice token embs Long, cross-referencing docs
Markdown-aware Heading-aware with breadcrumb Structured docs
Code-aware AST per function or class Source code

Context preservation patterns#

  • Overlap: cheapest, default 15%
  • Sliding window: smaller stride, more chunks
  • Parent-child: retrieve small, serve big to LLM

Failure modes#

  • Table mid-cut (keep tables intact)
  • Function mid-cut (use AST)
  • Acronym at boundary (use overlap)
  • Tiny chunks dominating top-K (set min size)
  • Reindex on chunker change (dual-index cutover)

Picker#

Content Chunker
FAQ Recursive 256-512 + overlap
Books, long prose Semantic or late
Markdown site Markdown-aware + breadcrumb
Source code AST per function
Tables Per row, not text

Eval#

  • Build labeled (question, ground-truth chunk) set
  • Sweep size and overlap
  • Measure Recall@K, stop when plateau

Refs#

  • LangChain RecursiveCharacterTextSplitter
  • Jina, Late Chunking (2024)
  • tree-sitter (multi-language ASTs)
  • Greg Kamradt, semantic chunking writeups

FAQ#

What is the best chunk size for RAG?#

Most production RAG systems use 256 to 1024 token chunks with 10 to 20 percent overlap. Tune by measuring recall at top-K on a golden set, since the optimum depends on document length and query style.

What is semantic chunking?#

Semantic chunking splits documents at points where sentence embeddings show a similarity drop. This keeps coherent ideas together instead of cutting on a fixed token boundary, which improves retrieval precision.

Why do you need chunk overlap in RAG?#

Overlap bridges chunk boundaries so a sentence split in half still appears intact in at least one chunk. Without it, key facts that straddle a boundary often disappear from retrieval.

What is late chunking?#

Late chunking embeds the full document first using a long-context embedder, then slices the resulting token embeddings into chunk vectors. Each chunk vector carries whole-document context, which boosts recall on tricky queries.

Should I use recursive or fixed-size chunking?#

Recursive chunking respects paragraph and sentence boundaries by splitting on \n\n, \n, period, and space in order. It almost always beats fixed-size splitting on prose documents at the same chunk budget.