Embedding Pipeline at Scale#

An embedding pipeline for RAG indexing turns raw documents into a query-ready vector index. At scale it has five stages: ingest, chunk, embed in batches, index, and update incrementally. The hard parts are not the embedding model; they are versioning, deduplication, and backfill.

flowchart LR
  S[(Sources<br/>S3, DBs, APIs)] --> I[Ingest<br/>parsers + queue]
  I --> CH[Chunk<br/>semantic splits]
  CH --> DD[Dedupe<br/>content hash]
  DD --> EM[Embed<br/>batched, GPU]
  EM --> IDX[(Vector index<br/>HNSW or IVF)]
  EM --> META[(Metadata DB<br/>doc id, version, tenant)]
  S2[(Change events)] --> I

    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class S,S2 storage;
    class I,CH,DD service;
    class EM compute;
    class IDX,META datastore;

Ingestion pulls from heterogeneous sources: S3 PDFs, Confluence, Notion, internal databases, public web. A queue (Kafka, SQS) makes the stages async and replayable.

Chunking splits documents into 200 to 800 token segments. Semantic boundaries (paragraphs, headings) beat fixed-length cuts. Overlap (10 to 15 percent) preserves context across boundaries.

Batched embedding is where cost lives. A single GPU embedding model processes 5K to 50K texts per second when fed batches of 64 to 256. Sequential calls run 100x slower.

Incremental updates trigger off change events: a doc changes, you re-chunk it, re-embed only the changed chunks, and upsert into the index. Full re-embed only happens on model change.

Embedding versioning is non-negotiable. When you upgrade text-embedding-3-small to -large, the new vectors are not comparable to the old. Tag every vector with embed_model_version; backfill in a shadow index, switch atomically.

Problem statement

You ingest 50 million documents across 10K tenants, totaling 1 billion chunks. Documents change at ~1 percent per day. The embedding model will be upgraded once a year. Design an embedding pipeline for RAG indexing that costs under $50K per month at steady state, processes a doc edit within 60 seconds, and survives a full re-embed without downtime.

End-to-end architecture#

flowchart TB
  S1[S3 docs] --> CDC[CDC + change events]
  S2[Confluence] --> CDC
  S3[DB tables] --> CDC
  CDC --> Q[(Kafka:<br/>raw docs)]
  Q --> P[Parser workers<br/>PDF, HTML, MD]
  P --> Q2[(Kafka:<br/>parsed text)]
  Q2 --> C[Chunker<br/>semantic split]
  C --> H[Content hash<br/>dedupe]
  H --> Q3[(Kafka:<br/>chunks)]
  Q3 --> B[Batch builder<br/>collate to 256]
  B --> E[Embedder GPU pool<br/>vLLM-style batching]
  E --> U[Upsert]
  U --> IDX[(Vector DB<br/>Qdrant, Pinecone)]
  U --> META[(Postgres:<br/>doc, chunk, version)]

    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class S1,S2,S3 storage;
    class CDC,P,C,H,B,U service;
    class E compute;
    class Q,Q2,Q3 queue;
    class IDX,META datastore;

Five stages, each backed by a queue so failures are localized and retryable.

Stage 1: Ingestion#

Change Data Capture (CDC) from source systems. Approaches:

Source	Mechanism
S3	EventBridge or S3 Notifications
Postgres	Logical replication (Debezium)
Confluence / Notion	Webhook + periodic full scan
Salesforce, etc.	API polling + last-modified timestamp
Web	Scheduled crawler with sitemap

Each event lands on a Kafka topic with the doc URL or content, source ID, timestamp, and operation (create / update / delete).

Stage 2: Parsing#

def parse(raw):
    if raw.mime == "application/pdf":
        return pdfplumber_extract(raw.bytes)
    if raw.mime == "text/html":
        return readability_extract(raw.bytes)
    if raw.mime == "application/json":
        return json_to_text(raw.bytes)
    return raw.bytes.decode("utf-8", errors="ignore")

Hard cases: scanned PDFs (run OCR), tables (preserve structure with Markdown), images (caption with vision model). Failures go to a dead-letter topic for human review.

Stage 3: Chunking#

flowchart LR
  D[Document] --> H[Detect headings]
  H --> SS[Split at headings]
  SS --> S[Semantic split<br/>200-800 tokens]
  S --> O[Overlap 10-15%]
  O --> CC[Chunks with metadata]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class D,H,SS,S,O,CC service;

Chunk strategies:

Strategy	Pro	Con
Fixed length	Simple, fast	Cuts mid-sentence
Sentence-based	Respects sentences	Variable length
Semantic (LangChain RecursiveSplitter)	Respects structure	Slower
Late chunking (embed then chunk)	Best retrieval quality	Compute heavy
LLM-based (Anthropic contextual chunks)	Highest quality	Expensive

For most RAG, recursive splitter + 400 token chunks + 15% overlap is the workhorse. Each chunk inherits parent doc metadata (id, source, tenant, timestamps).

Stage 4: Deduplication#

def dedupe(chunk):
    h = sha256(chunk.text.encode()).hexdigest()
    existing = meta_db.find(content_hash=h, tenant_id=chunk.tenant_id)
    if existing:
        # New ref to existing vector
        meta_db.add_reference(existing.vector_id, chunk.doc_id)
        return None  # Skip embed
    return chunk

A 200-word boilerplate footer in every doc embeds once, references many. At scale this saves 20 to 40 percent of embed cost. Use MinHash or SimHash for near-duplicate detection on noisier corpora.

Stage 5: Batched embedding#

This is the GPU bottleneck and the place to optimize.

async def embed_loop(in_queue, out_queue):
    batch = []
    while True:
        try:
            chunk = await asyncio.wait_for(in_queue.get(), timeout=0.05)
            batch.append(chunk)
        except asyncio.TimeoutError:
            pass
        if len(batch) >= 256 or (batch and out_queue.size > 10000):
            texts = [c.text for c in batch]
            vecs = await embed_model.embed(texts)  # one GPU call
            for c, v in zip(batch, vecs):
                await out_queue.put((c, v))
            batch = []

Throughput on one A100 with bge-large-en: roughly 30K texts per second at batch 256, 3K per second at batch 1. Always batch.

Model	Dim	Throughput (A100, batch 256)	$/1M tokens (API)
OpenAI text-embedding-3-small	1536	n/a (API)	$0.02
OpenAI text-embedding-3-large	3072	n/a (API)	$0.13
Cohere embed-v3	1024	n/a (API)	$0.10
BAAI bge-large-en (self-host)	1024	~30K/s	~$0.005 effective
Voyage voyage-3	1024	n/a (API)	$0.12

Crossover: above ~50M tokens/day, self-hosting BGE / Nomic / GTE on A10G or A100 saves money vs API. Below that, APIs are simpler.

Indexing#

Vector DB options at scale:

DB	Strength	Watch out
Qdrant	Fast, self-host, payload filters	Memory bound at >100M vectors
Pinecone	Managed, scales to billions	Cost at scale
Weaviate	Hybrid (BM25 + vector)	Operational complexity
pgvector	Postgres-native, simple ops	Slower at 100M+ vectors
LanceDB	Cheap object store backing	Newer, less battle-tested
Milvus	Open source, billion-scale	Heavy ops footprint

HNSW is the default index. IVF + PQ trades recall for memory at huge scale. For 1B vectors, two-stage: IVF for shortlist (1000s), HNSW for top-k.

Incremental updates#

sequenceDiagram
  autonumber
  participant SRC as Source
  participant CDC as CDC
  participant PIPE as Pipeline
  participant IDX as Vector index
  participant META as Metadata DB
  SRC->>CDC: doc updated
  CDC->>PIPE: event(doc_id, new_content)
  PIPE->>PIPE: parse, chunk
  PIPE->>META: list old chunks for doc_id
  META-->>PIPE: [chunk_hash_1, hash_2, ...]
  PIPE->>PIPE: diff old vs new chunks
  PIPE->>PIPE: embed only new/changed
  PIPE->>IDX: upsert new vectors
  PIPE->>IDX: delete removed chunk vectors
  PIPE->>META: update doc version

The key trick: a doc edit usually changes 1 to 5 percent of chunks. Re-chunk the whole doc, compute hashes, and only embed the chunks whose hash changed. This cuts steady-state embed cost by 95 percent.

Embedding versioning and backfill#

flowchart LR
  V1[(Index v1<br/>embed model A)] --> Q[Live queries]
  N[New model B announced] --> S[Provision Index v2]
  S --> B[Backfill embed<br/>model B over corpus]
  B --> S2[Shadow query<br/>compare recall]
  S2 --> OK{Recall + cost OK?}
  OK -- yes --> CUT[Atomic cutover]
  OK -- no --> FIX[Tune chunking, retry]
  CUT --> V2[(Index v2 live)]
  V2 --> Q

    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class V1,V2 datastore;
    class Q,S,S2,OK,CUT,FIX service;
    class B,N compute;

You cannot mix vectors from two models in one index: their cosine similarities are not comparable. Strategy:

Stand up a parallel index v2
Run a backfill worker pool: read every chunk, embed with model B, write to v2
Once backfill is at 100% and shadow recall is good, cut traffic over atomically
Keep v1 hot for 24-48 hours for rollback
Tear down v1

Backfill cost for 1 billion chunks at 500 tokens each = 500B tokens. At $0.02/1M (OpenAI small) that is $10K, self-host BGE is ~$2K of GPU. Plan once a year.

Per-tenant partitioning#

Two patterns:

Namespace per tenant (Pinecone style): logical partition inside one physical index. Cheap, fast filter.
Index per tenant: hard isolation, separate ops. Required for the most sensitive customers.

For 10K tenants with skewed sizes (most tiny, a few huge), hybrid: shared index for the long tail with namespace filter, dedicated index for the top 10 by size.

Cost math#

Steady state for 1B chunks, 1% daily churn:

Daily new embed: 10M chunks * 400 tokens = 4B tokens
API cost (OpenAI small): 4B * $0.02 / 1M = $80/day = $2400/month
Self-host (1 A10G GPU + ops): ~$700/month
Vector DB (Pinecone, 1B vectors, 1024 dim): roughly $20K-40K/month
Storage + Kafka + ops: ~$5K/month

Vector DB dominates. Self-hosting Qdrant on dedicated hardware drops the index cost by 5-10x.

Production failure modes#

Failure	Cause	Fix
Index out of sync	Embed succeeded, upsert failed	Two-phase commit: stage in object store, ack only after upsert
Stale vectors after doc delete	Delete event missed	Periodic reconcile: list all chunk IDs in meta vs index
Embedding skew after model patch	Provider quietly upgraded	Pin embed model version explicitly, alert on response_model
Backfill stuck	One bad doc crashes worker	Per-doc try/except, DLQ
GPU underutilized	Batch too small, latency-tuned	Increase batch size, accept higher per-doc latency
Tenant leak	Filter not applied	Test in CI: query tenant A must never return tenant B chunks

Quick reference#

Stages#

Ingest (CDC events from S3, DBs, APIs)
Parse (PDF/HTML/MD/OCR)
Chunk (semantic, 200-800 tok, 10-15% overlap)
Dedupe (content hash)
Batched embed (GPU pool, 256 batch)
Upsert to vector DB + metadata DB

Chunk strategies#

Strategy	Use
Fixed length	Quick baseline
Recursive (LangChain)	Default workhorse
Sentence-based	Respect natural splits
Late chunking	Best quality, more compute
LLM contextual chunks (Anthropic)	Premium quality, $$$

Embedding throughput#

Setup	Batch 256 throughput
API OpenAI / Cohere	RPS limited
BGE on A100 self-host	~30K texts/s
BGE batch 1	~3K/s (10x worse)

Always batch. Build a 50ms collation window.

Embed model crossover#

<50M tokens/day: API simpler
50M tokens/day: self-host BGE / Nomic / GTE pays off

Incremental update trick#

Re-chunk full doc
Hash each chunk
Embed only chunks whose hash changed
Steady state: 95% cost reduction vs full re-embed

Versioning + backfill#

New model -> new index v2
Backfill workers re-embed entire corpus
Shadow query v1 + v2, compare recall
Atomic cutover when recall + cost OK
Keep v1 hot 24-48h for rollback
Annual event for most teams

Never mix vectors from two embed models in one index.

Per-tenant partitioning#

Namespace (Pinecone): logical partition, cheap filter
Index per tenant: hard isolation, more ops
Hybrid for 10K+ tenants with skew

Vector DBs#

DB	Fit
Qdrant	Self-host default
Pinecone	Managed, billion-scale
Weaviate	Hybrid BM25+vector
pgvector	Simple, small-scale
LanceDB	Object-store backing
Milvus	Open billion-scale

HNSW default, IVF+PQ at >100M for memory.

Steady-state cost (1B chunks, 1% churn/day)#

Daily new embed: ~4B tokens
API embed: $2400/mo, self-host $700/mo
Vector DB managed: $20-40K/mo
Self-host Qdrant: ~$4K/mo Vector DB dominates; self-host saves 5-10x.

Failure modes#

Embed-upsert split brain -> two-phase via object store
Stale after delete -> periodic reconcile
Provider model drift -> pin response model
Backfill stuck -> per-doc try/except + DLQ
Tenant leak -> CI test forbids cross-tenant returns
GPU under-used -> raise batch size, tolerate latency

Refs#

Anthropic contextual retrieval (2024)
BAAI BGE model card; Nomic Atlas; GTE
Qdrant, Pinecone, Weaviate, pgvector docs
LangChain RecursiveCharacterTextSplitter

FAQ#

How does an embedding pipeline work?#

It ingests raw documents, chunks them, deduplicates by content hash, embeds the chunks in GPU batches, and writes vectors plus metadata to the vector index. A change stream keeps the index incremental.

How do you handle re-embedding when the model changes?#

Version every chunk by embedding model id. On a model change, write a new index alongside the old one, backfill in the background, and switch reads atomically once recall metrics meet the threshold.

How do you deduplicate documents before embedding?#

Hash the chunk text after normalisation and check the hash against a metadata store. Skip embedding if the hash already exists, which saves both GPU minutes and vector index storage.

What batch size is best for embedding throughput?#

Modern GPU embedders saturate around 64 to 256 chunks per batch depending on token length. Measure tokens per second per dollar and pick the batch size that maximises it without hitting OOM.

Should I use HNSW or IVF for the vector index?#

HNSW gives the best latency and recall for in-memory workloads. IVF or IVF-PQ wins when the index is too large to fit in RAM and you need on-disk search with quantisation.

RAG Patterns: how the indexed embeddings are consumed
Chunking Strategies: deeper dive on chunk boundaries and overlap
Vector Databases: index data structures (HNSW, IVF, PQ)

Embedding Pipeline at Scale#

End-to-end architecture#

Stage 1: Ingestion#

Stage 2: Parsing#

Stage 3: Chunking#

Stage 4: Deduplication#

Stage 5: Batched embedding#

Indexing#

Incremental updates#

Embedding versioning and backfill#

Per-tenant partitioning#

Cost math#

Production failure modes#

Quick reference#

Stages#

Chunk strategies#

Embedding throughput#

Embed model crossover#

Incremental update trick#

Versioning + backfill#

Per-tenant partitioning#

Vector DBs#

Steady-state cost (1B chunks, 1% churn/day)#

Failure modes#

Refs#

FAQ#

How does an embedding pipeline work?#

How do you handle re-embedding when the model changes?#

How do you deduplicate documents before embedding?#

What batch size is best for embedding throughput?#

Should I use HNSW or IVF for the vector index?#

Related Topics#