Exact duplicate rows in a training set are easy to catch with a simple hash check. Near-duplicates — the same underlying example paraphrased slightly, or scraped from two sources — are far more common and much harder to catch, and they bias training just as much.
Why near-duplicates matter
If forty of your five hundred examples are all near-identical, the model effectively sees that pattern eight percent of the time treated as forty percent, distorting what it learns to prioritise.
Detecting them
- Compute embeddings for each training example and flag pairs above a high similarity threshold for manual review.
- Check for examples sourced from the same origin document or conversation, which often produce near-duplicates even after individual cleaning.
- Cap how many near-duplicate examples of any one pattern survive into the final set, rather than removing all but keeping the count arbitrary.
See dataset validation before training.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.