Most student projects have 200 to 2,000 labelled examples, not the tens of thousands a paper assumes. That's enough to teach a narrow behaviour — it is not enough to teach new knowledge or broad reasoning.
What small datasets teach well
- A consistent output format (always reply in this JSON shape, always this tone).
- A narrow classification task with a handful of clear labels.
- A style or persona shift on top of capability the base model already has.
What they can't teach
New factual knowledge the base model doesn't already have, or a reasoning skill it genuinely lacks. For those, RAG or a bigger base model is the right tool, not more fine-tuning epochs on a small set.
See RAG vs fine-tuning: when to use which for that exact decision, and instruction tuning vs domain adaptation for what each small dataset is realistically shifting.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.