The dataset is the fine-tune.

Example dataset for fine tuning LLM

The common format is JSON Lines: one example per line, each a JSON object with a messages list containing a system message (optional), a user message with the input, and an assistant message with the ideal output. For a support-reply fine-tune a line holds the customer's message as the user turn and the reply you wish an agent had written as the assistant turn. For extraction, the user turn is the document text and the assistant turn is the JSON you want back. Keep every line in the same shape, use the model's own chat template at training time, and keep a separate file of held-out examples the model never sees.

Example LoRA training dataset

A LoRA dataset is no different from any supervised fine-tuning dataset; LoRA changes how the model is trained, not what it is trained on. A typical small one has three hundred to two thousand examples. For a classification task: input text and a single label word. For a style task: a neutral draft and the same content rewritten in the target voice. For a format task: unstructured input and a strictly valid JSON output. Public examples on Hugging Face include instruction datasets such as Alpaca-style and Dolly-style collections, useful for understanding the shape, less useful for your task, which needs your inputs.

What is synthetic data?

Synthetic data is data produced by a program or a model rather than collected from the real process you care about. For language model fine-tuning it usually means examples drafted by a strong model from a prompt: generate twenty customer questions about refunds, generate the ideal reply to each, vary the tone and the edge cases. It can also mean programmatically generated examples, such as templated inputs with known correct outputs for extraction tasks. It is fast and cheap, and it carries the biases and blind spots of whatever generated it, which is why it needs a human in the loop.

How is synthetic data generated?

• Seeded generation: give a strong model a handful of real examples and ask for more in the same style, varying one attribute at a time.
• Persona or scenario prompting: generate inputs from different user types and situations to widen coverage.
• Self-instruct and evol-instruct: let a model propose tasks, then answers, then harder variants of those tasks.
• Template filling: write input templates with slots and fill them programmatically with known values, so the correct output is known by construction.
• Distillation: use a large model's answers to real inputs as targets for training a smaller one.
• Augmentation: paraphrase, translate and back-translate, or perturb real examples to create variants.

How to create synthetic data in Python

Write a script that loads a small set of real seed examples, builds a generation prompt with instructions and a few seeds, calls a model API (or a local model through Ollama) in a loop with varied scenario parameters, and parses each response into the dataset format with validation against a schema. Deduplicate near-identical outputs with embedding similarity, filter by rules (length, required fields, banned content), and write to JSON Lines with a provenance field marking the example as synthetic. Then sample a hundred at random for a person to review and correct. Libraries such as distilabel package this workflow; a hundred-line script does the same and is easier to understand.

When to use synthetic data

Use it when real examples are scarce, when rare but important cases are under-represented (errors, edge cases, adversarial inputs), when real data cannot be used for privacy reasons, or when the correct output can be computed from a template so the labels are guaranteed right. Do not use it as the whole dataset for a task where the real inputs have a distribution you do not understand; a model fine-tuned only on synthetic customer questions learns the generator's idea of customers, not yours. My rule on FineTune Studio projects: real inputs wherever possible, synthetic outputs drafted and then edited by a person, synthetic inputs only to fill named gaps.

Synthetic data vs real data

Real data carries the true distribution: the phrasing, the mess, the long tail of cases you did not anticipate. It is slow to collect and may be private. Synthetic data is fast, unlimited and clean, and that cleanliness is the problem: it is smoother and more regular than reality, so a model trained on it alone over-fits to the generator's style and stumbles on real inputs. The strongest datasets mix them: real inputs with carefully written outputs as the core, synthetic examples to cover gaps, and a held-out set made only of real data so evaluation reflects what the model will actually see.

Why use synthetic data?

Because it removes the cold-start problem. On day one you may have ten real examples; a strong model can draft two hundred plausible ones in minutes, a person can correct them in an afternoon, and you have a first fine-tune to learn from. Because it lets you target weaknesses: when evaluation shows the model fails on a category, you generate more of that category. Because it protects privacy: you can train a model on synthetic records that resemble real ones without exposing anyone. And because it is cheap to iterate: regenerating a slice of the dataset costs minutes, where collecting it would cost weeks.

What can synthetic data be used for?

• Bootstrapping a fine-tuning set before real data exists.
• Covering edge cases, adversarial inputs and refusals that real logs rarely contain.
• Building evaluation sets with known correct answers, when generated from templates.
• Training small models by distilling a larger one's outputs on real inputs.
• Augmenting a small real dataset with paraphrases to improve robustness.
• Sharing a dataset publicly or across teams without exposing private records.
• Testing pipelines end to end with realistic volume before real traffic arrives.

How to generate synthetic data

Start from the gap, not from the generator. List the input types and edge cases your evaluation shows are missing. For each, write a generation prompt with three real seeds, explicit variation instructions and the exact output format. Generate in small batches, validate against a schema, deduplicate, and tag provenance. Review a random sample by hand and fix the generator prompt when you see systematic problems (over-polite tone, repeated structures, impossible details). Keep synthetic examples out of the held-out set. Re-run evaluation after adding them; if the real-data score did not move, the synthetic slice is not helping and should be cut rather than kept.