A student fine-tuned a model hard on customer-support replies and it got noticeably worse at basic reasoning questions it had handled fine before. That's catastrophic forgetting — narrow, aggressive fine-tuning overwrites general capability the base model had.
Why it happens
Too many epochs on too narrow a dataset pushes the model's weights hard in one direction. LoRA reduces this risk versus full fine-tuning because it touches far fewer parameters, but it isn't immune, especially at high rank and learning rate.
How to prevent it
- Evaluate on a general-capability set before and after fine-tuning, not just on the target task.
- Fewer epochs and a lower learning rate first; add more only if the target task genuinely needs it.
- Mix a small slice of general instruction data into the fine-tuning set alongside the task-specific data.
See reading a loss curve during fine-tuning for how forgetting often shows up as a training loss that looks fine while eval quality quietly drops elsewhere.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.