Retrieve wide, then re-rank narrow.

What are reranking models?

Re-ranking models are small transformer models trained to score the relevance of a passage to a query. Unlike an embedding model, which encodes the query and the passage separately, a re-ranker (a cross-encoder) reads both in one input and attends across them, so it can notice that the passage answers the question rather than merely being about the same topic. Popular ones include open cross-encoders from the BGE and MS MARCO families and hosted rerankers from Cohere and Voyage.

How reranking model works

The query and a candidate passage are concatenated with a separator and passed through the model, which outputs one relevance score. This is repeated for every candidate and the candidates are sorted by score. Because the model sees both texts at once it is much more accurate than vector similarity, but it is also slower, so it is only applied to the shortlist a cheaper retriever produced. Twenty to fifty candidates is the usual budget.

How to do reranking in RAG

• Retrieve a wide shortlist: twenty to fifty passages from hybrid search.
• Score each with a cross-encoder (sentence-transformers CrossEncoder, or a hosted rerank API).
• Keep the top three to five by score; drop anything below a floor if you use a confidence gate.
• Put only those into the prompt, in score order, with their sources.
• Measure answer faithfulness and recall at k before and after; keep the re-ranker only if the numbers move.

Budget the latency. A small cross-encoder scores about twenty pairs in well under a second on a CPU and in tens of milliseconds on a GPU; fifty pairs with a larger model can take a second or more, which users notice in a chat interface. Batch the pairs into one forward pass rather than scoring them one at a time, cache scores for repeated questions, and truncate passages to the chunk size you actually index so the model never sees padding. If latency is tight, re-rank the top ten instead of the top twenty and check whether recall at three moved. In RAG.NextUpgrad the re-ranker runs between hybrid fusion and the confidence gate, and its scores feed the gate, so the one model does two jobs.

What does reranking do in RAG?

It decides which passages the language model actually reads. First-stage retrieval optimises for recall: get the right passage somewhere in the top twenty. Re-ranking optimises for precision: put it in the top three. In RAG.NextUpgrad the fused hybrid results are re-ranked before the confidence gate looks at them, so the gate judges the best evidence rather than the noisiest, and the model's context stays short.

Why we need reranking in RAG

Because embedding similarity is coarse. A passage can be very close to the query in vector space while answering a neighbouring question, and the passage with the exact fact can sit at rank twelve. Sending twenty passages to the model to be safe costs tokens and buries the answer. A re-ranker lifts the right passage to the top so you can send five. It is the cheapest large improvement most RAG pipelines have left.

Reranking vs vector search

They are different stages, not rivals. Vector search is a bi-encoder comparison: fast, scalable, approximate. Re-ranking is a cross-encoder comparison: slow, precise, applied to a shortlist. You cannot re-rank a million documents, and you should not trust vector search to order the final five. The standard design is vector (or hybrid) search for candidates, then re-ranking for the final order.

Difference between embedding and reranking

An embedding model turns one text into a vector in isolation; relevance is then a distance between two vectors computed later. A re-ranking model takes two texts together and outputs a relevance score directly. The first lets you pre-compute and index; the second cannot be pre-computed because it depends on the query. That is why embeddings do the wide search and re-rankers do the narrow one.

How does semantic reranking work?

Semantic re-ranking is re-ranking with a model that understands meaning, which in practice is any modern cross-encoder. Some systems use a language model as the re-ranker: show it the query and the candidates and ask it to order them or to pick the ones that answer the question. That is accurate but expensive and slower; a dedicated cross-encoder gives most of the benefit at a fraction of the cost and latency.

Does Ollama support reranking?

Ollama is built for running generative and embedding models locally; it serves embedding models well, and re-ranking is best done with a cross-encoder through a library such as sentence-transformers or a small FastAPI service in front of it. You can also ask a local model served by Ollama to re-rank a short list in a prompt, which works for prototypes. For a production pipeline a dedicated cross-encoder is faster and more predictable.

Difference between retrieval and reranking

Retrieval searches the whole collection and returns a shortlist quickly; re-ranking reads only that shortlist and orders it carefully. Retrieval is where recall is won or lost; re-ranking is where precision is won. If the right passage is not in the retrieved twenty, no re-ranker can save you, which is why you measure recall at k first and only then tune the re-ranking stage.