Reranking#

RAG reranking cross-encoder pipelines run a fast vector search to recall a wide candidate pool, then a slow but accurate cross-encoder model re-scores the top results so the LLM sees the highest-precision context first.

A bi-encoder embeds query and document independently: cheap, ANN-friendly, but ignores fine-grained query-document interaction. A cross-encoder takes [query, doc] as one input and produces a relevance score that captures token-level attention. You pay for the precision in latency, so it only runs on the top 50-200 candidates.

flowchart LR
  Q([User query]) --> Enc[Bi-encoder<br/>embed]
  Enc --> ANN[Vector ANN<br/>top-100]
  ANN --> Rerank[Cross-encoder<br/>BGE / Cohere / Voyage]
  Rerank --> TopK[Top-5 by precision]
  TopK --> LLM[LLM with context]

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
  class Q client;
  class Enc,ANN service;
  class Rerank compute;
  class LLM external;

Why bother: raw ANN often ranks plausible-but-wrong chunks above the gold answer, especially on multi-hop questions or queries with negations. A reranker can lift Recall@5 from 60% to 90%+ on standard RAG benchmarks while keeping latency under ~300ms for 100 candidates.

Common rerankers:

BGE-reranker-v2 (BAAI): open-weights, self-host, MIT-friendly variants exist.
Cohere Rerank 3: hosted API, multilingual, ~100ms p50 for 100 docs.
Voyage rerank-2: hosted, tuned for code and finance domains.

The reranker is the cheapest precision win you can add to a RAG stack; almost every production RAG system uses one.

Problem statement

A RAG system over 5M chunks returns the top-20 by cosine similarity, but only 3 of the 20 are actually relevant. The LLM ends up grounded on noise and hallucinates. Design a stage that lifts precision without blowing the latency budget or running a cross-encoder over the full corpus.

RAG reranking cross-encoder models are the standard answer: a second, more expensive stage that re-scores a small candidate pool the bi-encoder produced. The pattern shows up everywhere from Perplexity to enterprise search to coding copilots.

Why bi-encoder retrieval is not enough#

A bi-encoder produces embed(query) and embed(doc) separately and compares with cosine. That makes ANN possible, but it also means the model never sees query and doc together. Subtle distinctions get lost:

Negation: "how to not invalidate JWT" vs "how to invalidate JWT".
Multi-hop: "the company whose CEO replaced Travis Kalanick".
Disambiguation: "Apple revenue" (fruit vs company).
Long tail jargon outside the encoder's training distribution.

Empirically, on MTEB and BEIR benchmarks, even strong bi-encoders like bge-large-en-v1.5 plateau around 50-55 nDCG@10, while a cross-encoder reranker on the same candidates pushes 65-72.

Two-stage retrieval#

flowchart TB
  Q([Query]) --> Bi[Bi-encoder<br/>embed query]
  Bi --> ANN[(Vector DB<br/>HNSW / IVF)]
  ANN -->|top 100| Pool[Candidate pool]
  Pool --> Cross[Cross-encoder reranker]
  Cross -->|rescored| K[Top-5]
  K --> Prompt[LLM prompt assembly]

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
  class Q client;
  class Bi service;
  class ANN datastore;
  class Pool edge;
  class Cross compute;
  class Prompt external;

Stage 1 (recall): bi-encoder + ANN returns the top 50-200 candidates in 10-30ms. Stage 2 (precision): cross-encoder scores each (query, doc) pair in a single forward pass. With a 100-doc pool and a small reranker model, total stage-2 latency lands at 100-300ms.

The split is fundamental: you cannot afford to run the cross-encoder over 5M chunks, and you cannot afford the precision loss of trusting raw cosine.

Cross-encoder models in practice#

Model	Host	p50 latency (100 docs)	Strengths
`BAAI/bge-reranker-v2-m3`	Self-host (HF)	80-200ms on A10G	Multilingual, open weights
`BAAI/bge-reranker-large`	Self-host	50-120ms on A10G	English, strong on BEIR
Cohere `rerank-english-v3.0`	Hosted API	80-150ms	Long context (4k), pay per call
Cohere `rerank-multilingual-v3.0`	Hosted API	100-180ms	100+ languages
Voyage `rerank-2`	Hosted API	100-200ms	Tuned for code and finance
MS MARCO `cross-encoder/ms-marco-MiniLM-L-6-v2`	Self-host	20-40ms	Cheap baseline, English only

A reasonable default: start with Cohere Rerank for time-to-value, then move to bge-reranker-v2-m3 once cost or latency forces it.

Sample reranker call#

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-large", device="cuda")

def rerank(query: str, candidates: list[dict], k: int = 5):
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)
    for c, s in zip(candidates, scores):
        c["rerank_score"] = float(s)
    candidates.sort(key=lambda c: c["rerank_score"], reverse=True)
    return candidates[:k]

The cross-encoder returns an unbounded raw logit. Higher is better but values are not directly comparable across queries; do not threshold a single score globally without calibration.

Latency cost and budget#

A RAG turn budget is usually 1.5-3 seconds end-to-end. Typical split:

Stage	Budget
Query rewrite (optional LLM call)	200-400ms
Bi-encoder embed + ANN	30-80ms
Rerank (top 100, batch on GPU)	100-300ms
LLM generation	800-2000ms

If rerank latency overshoots, lower the pool size from 100 to 50, or run rerank in parallel with prompt assembly when the LLM call dominates.

Score calibration#

Raw cross-encoder logits are not probabilities. For decisions like "does this question have any relevant documents at all?", calibrate:

Build a held-out set of (query, doc, relevant?) triples.
Fit a Platt scaler (logistic regression) on the raw scores.
Pick a threshold (e.g. p_relevant > 0.4) tuned to your precision-recall target.

Without calibration, a low-score top-1 still gets passed to the LLM and triggers a confident hallucination.

Ensemble and fusion strategies#

For high-stakes RAG, combine multiple signals before the reranker (or in place of it):

Reciprocal Rank Fusion (RRF): merge BM25 and vector rankings; cheap, no training.
Hybrid + rerank: RRF for stage 1 pool, cross-encoder for stage 2 (Microsoft, Vespa, Weaviate all support this).
Rerank ensemble: average scores from two different rerankers (open + hosted) when correctness matters more than cost.
LLM-as-reranker: use a small LLM with a structured prompt to score each doc; slow and expensive, but useful for niche domains where no cross-encoder is trained.

Production failure modes#

Stale corpus, fresh queries. Cross-encoder cannot rescue chunks whose embeddings predate a major content update; reindex or invalidate cache.
Domain mismatch. Generic rerankers underperform on legal, medical, code; fine-tune or pick a domain-tuned model (Voyage for code).
Latency tail. Cold-start GPUs add 5-10s; keep a warm pool, or use a hosted API with autoscaling.
Score drift after model swap. Treat the reranker model version as part of the eval contract; rerun the regression suite on every swap.
Recall ceiling. No amount of reranking can rescue a candidate pool that does not contain the answer. Always check Recall@100 before tuning rerank.

Quick reference#

TL;DR#

Bi-encoder ANN recalls 100 candidates fast. Cross-encoder reranks them with token-level attention. Pass the top 5 to the LLM.

Why#

Bi-encoders miss negation, multi-hop, disambiguation.
BEIR / MTEB jump: 50-55 nDCG@10 (bi) to 65-72 (rerank).
Recall@5 lifts from ~60% to ~90% on standard RAG sets.

Stages#

Stage	Model	Latency	Output
Recall	Bi-encoder + ANN	10-30ms	Top 50-200
Precision	Cross-encoder	100-300ms	Top 3-10

Common rerankers#

Model	Type	Notes
`bge-reranker-v2-m3`	Open, self-host	Multilingual default
`bge-reranker-large`	Open, self-host	English, fast
Cohere Rerank 3	Hosted	Easiest TTV, 4k context
Voyage rerank-2	Hosted	Code, finance tuned
`ms-marco-MiniLM-L-6`	Open	Cheapest baseline

Latency budget (RAG turn)#

Embed + ANN: 30-80ms
Rerank top-100: 100-300ms
LLM gen: 800-2000ms

Score calibration#

Raw logits are unbounded, not probabilities.
Platt-scale on a labeled set for thresholding.
Threshold for "no relevant doc found" before passing to LLM.

Ensemble patterns#

RRF (BM25 + vector) before rerank.
Two-reranker average for high stakes.
LLM-as-reranker for niche domains without tuned models.

Watch-outs#

Domain mismatch kills generic rerankers, fine-tune or pick domain-tuned.
Score drift after model swap, version pin and re-eval.
Recall@100 ceiling, no rerank can rescue an empty pool.
Cold-start GPU tail, keep warm pool.

Refs#

BGE reranker, BAAI HuggingFace org
Cohere Rerank 3 docs
Voyage AI rerank-2 docs
BEIR benchmark (Thakur et al, 2021)
Pinecone blog: two-stage retrieval

FAQ#

What is reranking in RAG?#

Reranking is a second pass over the top results from vector search where a cross-encoder model scores each query and document pair together. The most relevant documents move to the top before the LLM sees them.

What is the difference between a bi-encoder and a cross-encoder?#

A bi-encoder embeds query and document independently and compares vectors, which is fast and ANN-friendly. A cross-encoder feeds query and document together, which is more accurate but too slow for full search.

When should I add a reranker to RAG?#

Add a reranker when retrieval recall is fine but the LLM keeps grounding on the wrong chunks, or when top-1 precision matters. It usually lifts answer accuracy by 5 to 15 percentage points on benchmarks.

Which reranker model should I use?#

Cohere Rerank is the strongest commercial option. Among open models, BGE reranker, mxbai-rerank, and Jina rerank are competitive. Pick by latency budget, since cross-encoders scale linearly with candidate count.

How many candidates should I rerank?#

Most pipelines retrieve 50 to 200 candidates and rerank to a top 5 to 10 for the LLM prompt. More candidates lift recall but raise latency, so tune against an end-to-end answer-quality metric.

RAG Patterns: end-to-end retrieval pipeline that reranking plugs into
Vector Databases: ANN substrate that builds the bi-encoder candidate pool
Agent Memory: rerankers also tune semantic-memory retrieval at turn start

Reranking#

Why bi-encoder retrieval is not enough#

Two-stage retrieval#

Cross-encoder models in practice#

Sample reranker call#

Latency cost and budget#

Score calibration#

Ensemble and fusion strategies#

Production failure modes#

Quick reference#

TL;DR#

Why#

Stages#

Common rerankers#

Latency budget (RAG turn)#

Score calibration#

Ensemble patterns#

Watch-outs#

Refs#

FAQ#

What is reranking in RAG?#

What is the difference between a bi-encoder and a cross-encoder?#

When should I add a reranker to RAG?#

Which reranker model should I use?#

How many candidates should I rerank?#

Related Topics#