Vector search compares a query embedding to passage embeddings that were computed separately, so it never actually reads the query and the passage together. That is why it can rank a chunk about "refund policy for cancelled orders" above the one about "refund timelines" when a user asks how long refunds take. A reranker fixes exactly this gap.

What a cross-encoder does differently

A cross-encoder takes the query and one candidate passage as a single input and outputs a relevance score. Because it attends across both texts at once, it catches negations, specific entities and conditions that a bi-encoder blurs. The cost is that it must run once per candidate, so you never rerank the whole index — only the top 20 to 50 results from the first-stage retriever.

The two-stage pipeline

  1. Retrieve broadly: hybrid dense + BM25 search returns the top 30 candidates (see hybrid retrieval explained).
  2. Rerank precisely: score each candidate with the cross-encoder, keep the top 5.
  3. Gate: if the best reranked score is below a threshold, refuse instead of answering (see what a confidence gate is).
  4. Generate with citations from the surviving chunks only.

What it changed in RAG.NextUpgrad

Adding a reranker was the step that made the confidence gate trustworthy. Before it, first-stage scores were noisy enough that a threshold either blocked good answers or let weak ones through. Reranked scores separate relevant from irrelevant far more sharply, so the gate has something honest to measure. The platform still streams the answer token by token; the reranking pass adds a short pause before the first token, which users accept in exchange for fewer wrong answers.

Cost and latency, honestly

A small cross-encoder on CPU scores 30 passages in a few hundred milliseconds. On a free-tier server that is noticeable but fine; on GPU it is negligible. The memory footprint matters more than speed on small hosts — I covered that trade-off in the free-tier memory-budget playbook.

When to skip it

  • Your corpus is tiny and questions are simple — first-stage retrieval already puts the right chunk on top.
  • You have hard latency limits under 300 ms and no GPU.
  • Your evaluation set shows recall@5 above 95% without reranking. Measure before adding machinery.

If your RAG app gives confident wrong answers, add a reranker before you touch the prompt. In my experience it is the highest-leverage component after chunking.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.