RAG, explained by someone who runs one.

What is RAG in AI?

RAG stands for retrieval-augmented generation. Instead of asking a language model to answer from whatever it memorised during training, you retrieve relevant text from your own documents first and put that text into the prompt. The model then writes an answer grounded in what it was shown. The result is an assistant that knows your data, can be updated by adding a file, and can point to where an answer came from.

What does RAG stand for?

Retrieval-Augmented Generation. Retrieval is the search step that finds relevant passages; generation is the language model writing the answer; augmented means the generation is improved by what was retrieved. The term comes from a 2020 research paper by Lewis and colleagues at Facebook AI Research, but the pattern is now the default way to build question-answering over private documents.

What is retrieval augmented generation?

It is a two-stage system. Stage one indexes your documents: they are split into chunks, each chunk is turned into an embedding (a list of numbers that captures meaning) and stored in a vector index, often alongside a keyword index. Stage two answers a question: the question is embedded, the closest chunks are fetched, optionally re-ranked, and the top few are placed in the prompt with an instruction to answer only from them and to cite them.

How retrieval augmented generation works

In RAG.NextUpgrad, the system I built and ran in production, a question goes through five steps: it is embedded; dense (vector) search and sparse (BM25) search each return candidates; the two lists are fused with reciprocal-rank fusion; a re-ranker orders the fused list; and a confidence gate scores the evidence before the model is allowed to answer. If the evidence is weak the system says so rather than guessing. The answer streams back with inline citations to the exact passages.

A worked example makes it concrete. A student uploads a 300-page course handbook and asks when the scholarship form is due. The question is embedded and also tokenised; the vector index returns passages about deadlines and forms, the keyword index returns every passage containing 'scholarship'. The fused list is re-ranked, and the passage with the actual date rises to the top. The gate sees strong evidence, the model answers with the date and a citation to page 112, and the student can click through to check. Ask the same system about the canteen menu, which the handbook does not cover, and the gate finds no supporting passage and says so. That contrast, answer with proof or decline with a reason, is what retrieval-augmented generation buys you over a chat model on its own.

What is RAG pipeline?

A RAG pipeline is the chain of components between a raw document and a cited answer: loaders that read PDFs, HTML or databases; a chunker; an embedding model; a vector store and usually a keyword index; a retriever that combines them; a re-ranker; a prompt template; the language model; and the evaluation harness that tells you whether any change made answers better or worse. Treat it as a pipeline because every stage can be measured and swapped on its own.

How to build RAG pipeline

• Collect a few hundred real questions people actually ask, with the answers they expect. This is your test set and you build it first.
• Chunk documents by structure (headings, paragraphs) at a few hundred tokens with a small overlap; embed with a current open or hosted embedding model; index in pgvector, Qdrant or FAISS, plus BM25.
• Retrieve ten to twenty candidates from each index, fuse, re-rank to the top three to five.
• Prompt the model to answer only from the passages and to cite them; stream the answer.
• Measure recall at k for retrieval and faithfulness for answers on your test set after every change.

What is RAG chatbot?

A RAG chatbot is a chat interface in front of a RAG pipeline: each message is used as a retrieval query (often rewritten using the conversation so far), the retrieved passages are added to the prompt along with recent turns, and the reply cites sources. The difference from a plain chatbot is that it answers from your knowledge base and can decline when the knowledge base has nothing, which is exactly what a support or documentation assistant needs.

What is hybrid search in RAG?

Hybrid search runs two retrievers and merges their results: dense vector search, which matches meaning and paraphrase, and sparse keyword search such as BM25, which matches exact terms, product codes, names and numbers. Each fails where the other succeeds. Merging them with reciprocal-rank fusion is a dozen lines of code and, in my experience, fixes most of the bad answers a vector-only system produces.

What is reranking in RAG?

Re-ranking is a second, slower scoring pass over the candidates the first retrieval returned. A cross-encoder model reads the question and each passage together and scores how well that passage answers the question, which is far more accurate than comparing two embeddings. You retrieve twenty, re-rank, and keep the top five. The model then sees less text and better text, which improves answers and lowers cost at the same time.

How to evaluate RAG pipeline

Evaluate retrieval and generation separately. For retrieval, measure recall at k and mean reciprocal rank against your labelled question set: did the right passage appear in the top k? For generation, measure faithfulness (does every claim trace to a passage) and answer relevance, using a rubric or an LLM judge you have checked against human grading. Run the whole set in CI so a change to chunking or prompts cannot silently make things worse. RAG.NextUpgrad ships with a test suite for exactly this reason.