It's tempting to stuff the top twenty retrieved chunks into the prompt just because the context window allows it. Past a certain point, more retrieved text doesn't improve answer quality — it dilutes the model's attention and costs more tokens for no gain.
A practical budget
- Start with the top 3-5 chunks by relevance score, not an arbitrary large number.
- Measure answer quality on your eval set as you vary the count — most tasks plateau surprisingly early.
- Reserve a fixed token budget for retrieved context, separate from the budget for instructions and conversation history, so one doesn't crowd out the other unpredictably.
The 'needle in a haystack' problem
Models can miss relevant facts buried in the middle of a long context even when they're technically 'in the window' — this is a documented weakness, not a rare edge case. Fewer, better-chosen chunks often beat more, weaker ones.
See where a RAG request's latency actually goes.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.