A project accumulating new labelled data every month faces a choice: retrain from the original base model each time, or keep fine-tuning the already-tuned model incrementally. The second is cheaper but carries a real risk.
The risk: compounding forgetting
Each additional round of fine-tuning on new data can erode capability learned in earlier rounds, especially if the new data's distribution differs even slightly from the old. This compounds silently across many rounds if never re-checked.
A safer approach
- Keep a fixed evaluation set spanning ALL prior rounds' concerns, not just the newest data, and check it after every continual fine-tune.
- Periodically retrain from the original base model on the full accumulated dataset, rather than purely incremental updates indefinitely.
- Version every checkpoint clearly so a regression can be traced to the specific round that caused it.
See catastrophic forgetting in fine-tuned models.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.