Duplicate and near-duplicate documents in a RAG index: the quiet quality killer

The same policy document uploaded three times with minor edits crowds out other results and confuses retrieval. How to detect and handle near-duplicates.

Written by

Pranjul Rathour

GenAI Engineer at SCULT INDIA, based in Kanpur, India. Builds production RAG systems, fine-tunes open LLMs and ships agentic AI products; has mentored 200+ college developers. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks.

pranjulrathour41@gmail.com LinkedIn GitHub X Instagram Bluesky Dev.to

Presenting to a room

A document set that's grown over time often contains the same content uploaded multiple times with small edits — three versions of the same policy, two copies of the same manual from different folders. These near-duplicates crowd retrieval results and confuse which version is authoritative.

Detecting near-duplicates

Compare document-level embeddings for high similarity (not just exact-text hashing, which misses reworded duplicates).
Flag documents with near-identical titles or matching metadata (same source, close dates) for manual review.
Check retrieval results periodically for the same underlying fact appearing from multiple near-duplicate sources — that's a strong signal deduplication is needed.

Handling them

Keep the most recent or most authoritative version, and either remove or explicitly mark older duplicates as superseded rather than silently leaving them competing for retrieval slots.

See metadata filtering in RAG pipelines for the related date/source filtering this builds on.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.