Vector Search / RAG Retrieval

Problem statement (interviewer prompt)

Design a retrieval-augmented generation (RAG) service: ingest documents, chunk and embed them, store in a vector DB, and at query time retrieve the top-K most relevant chunks within 100ms to ground an LLM's answer.

flowchart TB
  Docs[(Source documents)] --> Ingest[Ingest worker<br/>chunk + embed]
  Ingest --> Vec[(Vector DB)]
  Ingest --> Meta[(Metadata store)]
  Q([User query]) --> API[Retrieval API]
  API --> Enc[Encoder]
  Enc --> Vec
  Vec --> Rerank[Re-ranker cross-encoder]
  Rerank --> Ctx[Prompt assembler]
  Ctx --> LLM[(LLM service)]
  LLM --> Resp([Answer])

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Q,Resp client;
    class API edge;
    class Ingest,Enc,Rerank,Ctx compute;
    class Vec,Meta,Docs datastore;
    class LLM external;

Two paths: ingest (chunk → embed → upsert) and retrieve (embed query → ANN → optional rerank → prompt LLM).

RAG is the pattern that grounds LLM answers in actual data: retrieve relevant context, then prompt the model to answer using only that context. The retrieval system is the half that breaks first.

Architecture

flowchart TB
  subgraph Ingest[Ingest path]
    Src[(Source: S3 / API / DB)] --> CDC[CDC / scheduled crawl]
    CDC --> Chunk[Chunker]
    Chunk --> EncBatch[Batch embedder]
    EncBatch --> Up[Upsert worker]
    Up --> Vec[(Vector DB Pinecone Weaviate pgvector)]
    Up --> Meta[(Metadata: chunk_id, doc_id, tenant, tags)]
  end
  subgraph Query[Query path]
    Q([User query]) --> API[Retrieval API]
    API --> QCache[(Cache)]
    QCache -. miss .-> Enc[Embedder]
    Enc --> ANN[ANN search top-K]
    Enc -.parallel.-> BM[BM25 keyword]
    ANN --> Fuse[Reciprocal-rank fusion]
    BM --> Fuse
    Fuse --> Filter[Metadata filter: tenant, ACL]
    Filter --> RR[Re-ranker cross-encoder]
    RR --> Out([Top-N chunks])
  end
  Out --> Prompt[Prompt builder] --> LLM[(LLM)]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Q,Out client;
    class API edge;
    class CDC,Chunk,EncBatch,Up,Enc,ANN,BM,Fuse,Filter,RR,Prompt service;
    class QCache cache;
    class Vec,Meta,Src datastore;
    class LLM external;

Chunking strategies

Strategy	When
Fixed N tokens with overlap	Default; 512 / 1024 tokens, 50-token overlap
Sentence / paragraph	Preserves semantic boundaries
Recursive by structure	Markdown headers, code blocks
Sliding window with metadata	Per-section title, source URL

Bad chunking is the single biggest quality killer. Too small = no context; too big = noisy retrieval.

Embedding pipeline

• Batch embed (32-128 per call) to amortize GPU cost.
• Idempotent: chunk_id is hash(doc_id, chunk_index, version).
• Re-embed on model change is a full re-ingest; version embeddings.

Hybrid search

final_score = α × vector_similarity + β × bm25_score
or
RRF(rank_vector, rank_bm25)

Vectors capture paraphrase; BM25 captures exact identifiers (CVE-XXXX, error codes). Almost always better than either alone.

Re-ranking

flowchart LR
  Vec[Top-100 from ANN] --> CE[Cross-encoder bge-reranker / cohere-rerank]
  CE --> Top10[Top-10 by joint relevance]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class CE compute;
    class Vec,Top10 datastore;

Cross-encoder is slower per pair but much more accurate. Apply to top-100 retrieval to get top-10 prompted.

Multi-tenant isolation

• tenant_id in every chunk's metadata.
• Per-tenant ANN index for hot tenants (filtered ANN is otherwise expensive).
• ACL filter applied before re-rank.

Freshness

Need	Approach
Hourly updates	CDC + batch re-embed
Sub-minute	Stream chunks via Kafka; embed worker reads
Real-time delete	Soft-delete tombstone in metadata; filter at query

Vector DBs are slow at delete; mark and sweep periodically.

Caching

• Query cache: hash of question → results, short TTL.
• Embedding cache: same query embedded once per minute (LRU).
• Prompt cache (LLM): provider-side caching (Anthropic, OpenAI both offer).

Capacity sketch

10M chunks × 1536-dim float32 = 60 GB raw vectors. - HNSW index: ~90 GB total memory. - ANN query p99: 20-50ms at this scale. - Re-rank top-100 with cross-encoder: 200-400ms on GPU. - End-to-end: 100ms (cached) / 500ms (rerank) / 1.5s (with LLM).

Evaluation

• Recall@K: of the relevant chunks, what fraction make top-K?
• NDCG: rank-weighted relevance.
• Hit rate: did at least one relevant chunk make top-N?
• End-to-end: rubric scoring of generated answers.

How RAG composes with fundamentals

flowchart TB
  RAG((Vector search<br/>/ RAG))
  VEC[Vector databases<br/>ANN substrate]
  SRCH[Search internals<br/>BM25 hybrid partner]
  CACHE[Caching strategies<br/>query + embed cache]
  CDC[Change Data Capture<br/>fresh ingest]
  VEC --> RAG
  SRCH --> RAG
  CACHE --> RAG
  CDC --> RAG

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class RAG service;
    class VEC,SRCH,CACHE,CDC datastore;

Glossary & fundamentals

Tag	Concept	What it is	Page
HLD	Vector databases	the ANN substrate	vector-databases
HLD	Search internals	BM25 hybrid partner	search-internals
HLD	Caching strategies	query and embedding caches	caching-strategies
HLD	Change Data Capture	source for fresh ingest	change-data-capture

Quick reference

Functional requirements

• Ingest documents, chunk + embed
• Retrieve top-K relevant chunks
• Hybrid keyword + vector
• Per-tenant isolation
• Sub-second freshness for new content

Non-functional

• Retrieval p99 < 100ms (no rerank), < 500ms (rerank)
• End-to-end < 2s including LLM
• Recall@10 > 90%

Components

• Ingest worker (chunker + embedder)
• Vector DB (Pinecone, Weaviate, pgvector, Qdrant)
• Metadata store (tenant, ACL, source)
• Encoder (embedding model)
• ANN service + BM25 service
• Re-ranker (cross-encoder)
• Prompt assembler + LLM service

Chunking

512 tokens with 50-token overlap is a safe default. Adapt to doc structure.

Hybrid (RRF)

Run vector + BM25 in parallel; fuse with reciprocal rank fusion.

Re-ranking

ANN top-100 → cross-encoder → top-10 for prompt.

Capacity (10M chunks @ 1536-d)

• ~60 GB raw vectors, ~90 GB HNSW
• ANN p99: 20-50ms
• Rerank top-100: 200-400ms on GPU

Evaluation

• Recall@K, NDCG, hit rate
• End-to-end rubric scoring

Trade-offs

• Bigger top-K → better recall, more rerank cost
• Hybrid > vector-only in nearly all benchmarks
• Re-ranker improves top-10 ordering dramatically
• Embedding model swap = full re-ingest

Refs

• Lewis et al. - RAG paper (2020)
• Anthropic - Contextual retrieval
• Pinecone learn series
• LangChain Retrieval module

FAQ

How does retrieval augmented generation (RAG) work?

Documents are chunked, embedded, and stored in a vector DB. At query time the query is embedded, the top-K nearest chunks are retrieved by ANN search, and they are stuffed into the LLM prompt.

How do you chunk documents for RAG?

Split by semantic boundaries (paragraph, section) with overlap of 50 to 200 tokens. Keep chunks 200 to 800 tokens so retrieval is precise but each result holds enough context to be useful.

HNSW vs IVF for vector search?

HNSW gives the best recall/latency tradeoff for in-memory data and is the default in most vector DBs. IVF is more memory-efficient for very large indices but needs tuning to match HNSW recall.

Why add a re-ranker on top of ANN?

ANN retrieves on cheap cosine similarity, which trades precision for speed. A cross-encoder re-ranker scores the top-K with full attention, pushing the truly relevant chunks to the top.

How do you keep RAG under 100ms?

Cache embeddings of frequent queries, keep the index in memory, parallelize encoder and ANN calls, and limit re-ranking to the top 20. Avoid network hops between encoder and vector DB.

Related Topics

• Vector Databases: the ANN storage layer
• Search Internals: the keyword half of hybrid search
• LLM Serving: the inference layer RAG feeds into

Further reading

• Paper - Lewis et al. - Retrieval-Augmented Generation (NeurIPS 2020)
• Blog - Anthropic - Contextual retrieval
• Doc - LangChain - Retrieval module
• Blog - Pinecone - RAG patterns