Every RAG tutorial picks a chunk size and moves on. I ran the same document set through 128, 256, 512 and 1024-token chunks, same embedding model, same twenty-question eval set, and measured recall@5 and answer faithfulness for each.
What moved and what didn't
Recall@5 rose from 128 to 512 tokens, then flattened. Past 512, chunks started mixing two unrelated ideas, and faithfulness dropped even though recall looked fine — the right chunk was retrieved, but it also carried a wrong neighbouring fact the model quoted.
The one variable that mattered more than size
Overlap mattered more than raw size. A 256-token chunk with 50-token overlap beat a 512-token chunk with no overlap, because fewer answers got cut across a chunk boundary.
- Start at 256–400 tokens with 15–20% overlap; it is a better default than 512 with none.
- Chunk by structure first (headings, paragraphs), then cap by token count — never split mid-sentence.
- Re-run your eval set after any chunking change; the effect is rarely intuitive.
See RAG systems, a complete beginner's guide for the pipeline this experiment ran on, and what a confidence gate is for what to do when a chunk sits at the edge of relevance.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.