Where a RAG request's latency actually goes (it isn't the LLM call)

Profiling a real RAG endpoint end to end: embedding, vector search, reranking, prompt assembly and generation each take a slice — and the slowest part surprises most people.

Written by

Pranjul Rathour

GenAI Engineer at SCULT INDIA, based in Kanpur, India. Builds production RAG systems, fine-tunes open LLMs and ships agentic AI products; has mentored 200+ college developers. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks.

pranjulrathour41@gmail.com LinkedIn GitHub X Instagram Bluesky Dev.to

Requirements gathering, on stage

Students assume the LLM call is the slow part of a RAG request. I profiled a real endpoint end to end and it usually isn't.

The breakdown

Query embedding: 20–80ms, cheap unless you're calling a hosted API per request.
Vector search: usually under 50ms for a well-indexed store, but grows fast on an unoptimised flat index.
Reranking: often the biggest slice — a cross-encoder over 20-50 candidates can take 200–500ms.
Generation: streaming hides most of this from the user, even though it is the largest raw time.

The one fix that helps most requests

Stream the generation token by token so perceived latency drops even when total time doesn't, and rerank fewer, better-filtered candidates rather than reranking everything the first-pass search returns.

See streaming LLM responses with FastAPI and SSE and reranking with cross-encoders.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.