Agentic RAG: retrieval the model decides on.

What does agentic RAG mean?

Agentic RAG means a retrieval-augmented system in which a language model acts as a controller rather than only a writer. The model receives the question, reasons about what information is needed, calls retrieval as a tool (possibly several times with different queries or against different indexes), reads the results, judges whether they are sufficient, and loops until it can answer or decides it cannot. The 'agentic' part is that control flow is chosen by the model at run time rather than fixed by the developer. It is the same pattern as any tool-using agent, with retrieval as the main tool.

Difference between RAG and agentic RAG

Plain RAG is a straight line: embed the question, retrieve, re-rank, generate. It is fast, cheap, predictable and easy to evaluate, and it covers most document question answering. Agentic RAG is a loop with decisions: the model may rewrite 'did we ever work with them' into 'contracts with Acme Ltd 2023 to 2025', may query the contracts index and then the email index, may notice the results conflict and retrieve again, and may decide the question is unanswerable. The loop buys coverage on complex questions and pays for it with latency, cost and variance. Build the straight line first and measure where it fails.

What is agentic RAG architecture?

The common architecture has a planner, tools, a memory and a verifier. The planner is a model prompt that reads the question and conversation and emits the next action. Tools are retrievers (vector, keyword, hybrid, graph, SQL, web search) and sometimes calculators or APIs, each with a clear description. Memory holds what has been retrieved so far and the agent's notes. The verifier checks whether the gathered evidence answers the question, often with a confidence score, and triggers another step or the final answer. Around it sit limits: a maximum number of steps, a token budget, timeouts, and logging of every action for evaluation.

What is agentic RAG pipeline?

A typical run looks like this. The question arrives. The planner classifies it: simple lookup, multi-part, or needs a non-document source. For a simple lookup it calls hybrid retrieval once and answers. For a multi-part question it splits it, retrieves for each part, and merges. If retrieval returns weak evidence, it rewrites the query (synonyms, expanded context, a hypothetical answer) and retries, up to a limit. If the question needs numbers from a database, it calls a SQL tool. Finally it drafts an answer with citations, checks each claim against the evidence, and either returns the answer or says what it could not find.

What is agentic RAG example?

A support assistant is asked: 'My order from last month arrived damaged, can I still return it and will shipping be refunded?' Plain RAG retrieves the returns policy and answers half the question. An agentic system splits it into three: the return window for damaged goods (policy index), whether shipping is refunded for damaged returns (policy index, different section), and when the order actually arrived (orders API). It retrieves each, notices the window depends on the delivery date it just fetched, and composes a complete answer with citations to both policy sections. Three retrievals, one tool call, one coherent reply.

When to use agentic RAG

• Questions that need several retrievals or several sources to answer.
• Users who phrase questions vaguely, where query rewriting measurably lifts recall.
• Mixed data: documents plus a database plus an API.
• Cases where a wrong answer is costly and a verification loop is worth the latency.
• Not for: FAQ-style lookups, latency-sensitive chat, or any system where you have not yet measured a fixed pipeline. Most products do not need it, and the ones that do need it for a minority of questions, so route only those to the loop.

How to implement agentic RAG

Start with a working hybrid RAG and a test set. Wrap retrieval as a tool with a precise description and typed inputs. Write a planner prompt that can choose among retrieve, rewrite-and-retrieve, call another tool, or answer, and cap it at a handful of steps. Add a verifier step that scores the evidence before the final answer; reuse the confidence gate if you have one. Log every step with its inputs and outputs. Then run the test set and compare against the fixed pipeline on accuracy, latency and cost per question. Keep the agent loop only for the question types where it wins, and route the rest straight through.

How to build agentic RAG with hybrid search

Hybrid search is the right retrieval tool to hand an agent, because an agent's rewritten queries range from exact identifiers to loose paraphrases and hybrid handles both. Expose one tool, search(query, filters, k), that runs BM25 and vector retrieval, fuses with reciprocal-rank fusion and re-ranks, returning passages with scores and sources. Give the agent the scores so it can judge whether to retry. Let filters narrow by document type, date or customer so the agent can scope a second attempt. In RAG.NextUpgrad the hybrid retriever and the confidence gate were built as reusable stages for exactly this kind of composition.

How to evaluate agentic RAG

Evaluate three layers. Outcome: final answer correctness, faithfulness to retrieved evidence, and correct refusals, on a labelled question set, compared with the fixed pipeline. Process: did the agent take sensible steps, how many, did it loop pointlessly, did it call the right tool; log traces and sample them for review. Cost: latency and tokens per question, with a distribution rather than an average, because agent loops have long tails. Add adversarial questions that tempt the agent to over-retrieve or to answer without evidence. An agentic system that is right more often but three times slower and twice as expensive needs that trade-off stated in numbers.

Agentic RAG vs Graph RAG

They answer different needs and combine well. Graph RAG changes what is indexed: entities, relationships and community summaries, which helps multi-hop and whole-corpus questions. Agentic RAG changes how retrieval is driven: a model plans and iterates, which helps multi-step and multi-source questions. You can have an agent that uses a graph as one of its tools. If your failures are 'the facts are in three documents and the link between them is an entity', look at a graph. If your failures are 'the question needed two searches and a database call', look at an agent. If you have not measured, measure first.