Recall@k and precision@k need labelled query-document relevance pairs, which almost no student project has on day one. Waiting to build that dataset before evaluating anything means shipping blind for weeks.
What works before you have labels
- Write ten to twenty questions you know the answer to, by hand, and check whether the right chunk shows up in the top results — a small manual eval set beats no eval set.
- Use an LLM to judge whether retrieved context is relevant to a query, as a rough automated proxy while you build a real labelled set over time.
- Log real user queries and periodically spot-check a sample manually — this is where your eventual labelled set should come from.
Building toward real evaluation
Every manual check you do can become a labelled example if you save it. Within a few weeks of normal use, a genuine evaluation set accumulates almost for free.
See RAG evaluation metrics: faithfulness and recall.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.