Measure retrieval and generation separately.

How to evaluate RAG

Start with a test set: a hundred or more real questions, each labelled with the passage that answers it and, where possible, a reference answer. Then measure the two stages on their own. Retrieval is judged by whether the labelled passage appears in the top k results. Generation is judged by whether the answer is supported by the retrieved passages and whether it actually answers the question. Track both over time; a change that helps one can hurt the other.

Size the set sensibly. Fifty questions will show you large effects such as switching to hybrid search; a few hundred are needed before a two-point change in recall means anything. Stratify across document types and question styles, including short keyword queries and long conversational ones, because a pipeline tuned on one style regresses on the other. Version the set alongside the code, record the retriever configuration with every run, and keep the raw per-question results, not only the averages: the questions that flip from pass to fail after a change are where the real diagnosis happens.

What is RAG evaluation?

RAG evaluation is the discipline of measuring a retrieval-augmented system the way users experience it, with numbers that move when quality moves. It replaces 'it seems better' with recall at five went from 0.71 to 0.84 and faithfulness held at 0.95. Without it, every change to chunking, embedding model, prompt or re-ranker is a guess, and regressions reach users before you notice. RAG.NextUpgrad ships with a test suite for this reason.

What are RAG evaluation metrics?

• Recall at k: the labelled passage is somewhere in the top k retrieved results.
• Mean reciprocal rank (MRR): how high the first relevant passage ranks, averaged.
• Precision at k: how many of the top k are relevant, which matters for context cost.
• Faithfulness: every claim in the answer is supported by a retrieved passage.
• Answer relevance: the answer addresses the question asked.
• Answer completeness: nothing important from the reference answer is missing.
• Refusal correctness: the system declines when the evidence is absent, and only then.

What is MRR in RAG evaluation?

Mean reciprocal rank measures where the first relevant passage lands. If it is at rank one the score is 1, at rank two it is 0.5, at rank four it is 0.25, and you average across questions. It rewards systems that put the right thing at the very top, which is what matters when only the top few passages go into the prompt. Recall at k tells you whether the answer is findable; MRR tells you whether it is found first.

What is faithfulness in RAG evaluation?

Faithfulness asks: is everything in the answer actually supported by the passages the model was given? An answer can be fluent and even correct while being unfaithful, if the model pulled the fact from its own memory instead of the evidence. To score it, split the answer into claims and check each against the context, either with human graders or with a language model judge whose grades you have validated against humans on a sample. Low faithfulness is the hallucination signal.

What is answer completeness in RAG evaluation?

Completeness compares the answer with a reference answer and asks what is missing. A faithful, relevant answer can still omit the one condition the user needed. Score it by listing the key points of the reference and checking each is present. It needs reference answers, which cost effort to write, so many teams score completeness on a smaller subset and faithfulness and recall on the full set.

How to create a RAG evaluation dataset from documents

Take real questions first: support tickets, search logs, questions colleagues actually asked. Where you have none, generate candidate questions per chunk with a language model, then have a person keep only the ones a real user would ask and fix the wording. Label the answering passage for each. Add questions the documents cannot answer, so you can test refusals. Aim for a hundred to start, spread across document types, and keep adding from production logs.

How to evaluate RAG retrieval

Run every test question through the retriever only and record the ranked list. Compute recall at 5 and at 20 and MRR against the labelled passages. Try one change at a time: chunk size, embedding model, hybrid versus vector-only, re-ranker on or off. In my experience the biggest jumps come from hybrid search and re-ranking, and the biggest regressions from chunking changes that looked harmless. Recall at 20 tells you whether a re-ranker can help at all.

How to evaluate RAG output

With retrieval fixed, score the answers. Use a rubric: faithful to the context, relevant to the question, complete against the reference, correctly refuses when there is nothing. A language model judge can apply the rubric at scale, but check it on fifty human-graded examples first and report agreement. Read the failures by hand; the categories you find (wrong passage chosen, right passage ignored, over-confident refusal) tell you which stage to fix next.

Best practices in RAG evaluation

• Build the test set before the pipeline; it is the specification.
• Evaluate retrieval and generation separately, and report both.
• Validate any LLM judge against humans before trusting it.
• Include unanswerable questions and score refusals.
• Run the suite in CI so a change cannot ship a regression.
• Refresh the set from production questions every month.