Small models: narrow tasks, done cheaply and privately.

What are small language models?

Small language models (SLMs) are transformer language models with far fewer parameters than the frontier models, typically from a few hundred million to around ten billion. Families such as Phi, Gemma, Qwen, Llama and Mistral all publish small variants, and they are trained with the same methods as large ones, often on carefully curated data to get more capability per parameter. They fit in a few gigabytes, run at usable speed on a laptop CPU or a phone, and fine-tune on a single consumer GPU. The trade is capability: they know less, reason less reliably and follow complex instructions less well than large models.

What are small language models good for?

• Classification and routing: ticket categories, intent detection, moderation flags.
• Extraction into a schema: fields from invoices, forms, emails; DocuLens AI pairs a model with strict validation for this.
• Rewriting and formatting: summaries of a fixed shape, tone adjustment, templated replies.
• Retrieval-augmented answers over a narrow corpus, where the passages carry the knowledge and the model only needs to read and cite.
• On-device assistants where privacy, offline use or latency rule out a hosted model.
• High-volume batch jobs where the per-token cost of a large model would be prohibitive.

Why are small language models preferred for enterprise deployment?

Four reasons recur in conversations with teams. Cost: a small model serves thousands of requests on one GPU, or on CPUs, for a fraction of large-model API spend. Privacy and control: the model runs inside the company's network, with a pinned version that never changes under them. Latency: a small model answers in a fraction of the time, which matters in interactive products and in pipelines with many calls. And specialisation: a small model fine-tuned on the company's own examples is often more consistent on that task than a general large model prompted to do it. The large model is kept for the hard minority of requests, with routing between them.

Why do small language models underperform?

Capacity. With fewer parameters a model stores less world knowledge, so it hallucinates more on facts it was never able to memorise; it handles fewer steps of reasoning before losing the thread; it is more sensitive to prompt wording; and it degrades faster under quantization. Small models also struggle with long contexts and with instructions that contain many constraints at once. The remedies are to give them the knowledge (retrieval), to narrow the task (fine-tuning on one job), to keep prompts simple and consistent, and to route anything that needs broad reasoning to a larger model. Used that way, their weaknesses rarely show.

How to use small language models

Run them locally with Ollama or llama.cpp for prototyping, or serve them with vLLM when many users arrive. Write prompts that are short and explicit, with one or two examples. Put knowledge in the prompt via retrieval rather than expecting the model to know it. Validate outputs mechanically where possible: parse the JSON, check the label is from the allowed set, reject and retry on failure. Measure on a labelled set before trusting one with a task, and compare with a large model on the same set to see what you are giving up. Where the gap is large, fine-tune the small model on the task; the gap often closes.

How to build small language models

Almost nobody should pretrain one; even a one-billion-parameter model needs hundreds of billions of tokens and serious compute to be useful. What you build is a specialised small model from an existing one: pick a small instruct model, collect a few hundred to a few thousand examples of your task, fine-tune with LoRA or QLoRA, and evaluate on held-out data against the base and against a large model. A second route is distillation: have a large model produce answers to your real inputs and train the small model on those pairs, which transfers much of the large model's skill on that task. FineTune Studio runs both routes.

What small language models are available?

The major families all publish small members, and the roster changes every few months: Microsoft's Phi series, Google's Gemma small variants, Alibaba's small Qwen models including coder versions, Meta's small Llama releases, Mistral's small models, and compact reasoning models distilled from DeepSeek. Hugging Face maintains lists and leaderboards filtered by size, and Ollama's library tags models by parameter count so you can pull one that fits. Choose by your task and your memory, shortlist three, and run them on thirty of your own prompts; the leaderboards do not know your task.

What is catastrophic forgetting in the context of LLMs?

Catastrophic forgetting is when fine-tuning a model on a new, narrow dataset overwrites the weights that supported its earlier abilities, so the model gains your task and loses general knowledge, instruction following, other languages or reasoning. It is a known property of neural networks trained sequentially on different data. For small models it is more severe, because they have less spare capacity: the same weights that hold general ability are the ones your fine-tune pushes on. You detect it by running general prompts on the tuned model and comparing with the base; a tuned model that fails simple unrelated requests has forgotten.

How to prevent catastrophic forgetting in LLMs

• Use parameter-efficient fine-tuning: freeze the base and train LoRA adapters, so the general weights cannot be overwritten.
• Keep the learning rate small and the epochs few; stop at the validation minimum.
• Mix a slice of general instruction data (ten to twenty per cent) into the task dataset so general behaviour keeps being rehearsed.
• Keep the adapter rank modest; larger ranks have more room to change behaviour broadly.
• Regularise towards the base model, as DPO does with its reference, when training on preferences.
• Evaluate general prompts alongside task prompts on every run, and keep the base model available for routing.

Does LoRA prevent catastrophic forgetting?

It reduces it substantially, without eliminating it. Because the base weights are frozen, the model's original abilities remain physically present; the adapter adds a low-rank change on top. Remove the adapter and the base is exactly as it was, which full fine-tuning cannot offer. But an adapter with high rank, a high learning rate and many epochs can still steer outputs far enough that general prompts get worse answers, because the adapter's change applies to every input, not only your task. Treat LoRA as the first line of defence, keep its settings modest, mix in general data, and measure forgetting rather than assuming it away.