From raw model to assistant in three stages.

What is instruction tuning in LLM?

Instruction tuning is supervised fine-tuning of a pretrained language model on examples of instructions paired with good responses: questions with answers, tasks with completions, conversations with helpful replies. Before it, a model continues text; ask it a question and it may produce more questions. After it, the model treats input as a request and produces a response in the expected form. It is the step that turns a base model into a chat or instruct model, and it uses the same mechanics as any fine-tune: pairs, a loss on the response tokens, a few epochs. Your task-specific fine-tune is instruction tuning on a narrow distribution.

What is instruction tuned model?

An instruction-tuned model is a base model that has been through that supervised stage (and usually a preference stage after it). Model families publish both: a base version for research and further pretraining, and an instruct or chat version for use. For almost any application you want the instruct version, because it follows prompts, respects a chat template with system and user roles, and declines or hedges appropriately. When you fine-tune, you normally start from the instruct version too, so you inherit its ability to follow instructions and add your behaviour on top, unless your task is so unusual that the base is a better starting point.

What is RLHF?

Reinforcement learning from human feedback is a method for training a model to produce outputs people prefer. Humans compare pairs of model responses and pick the better one; those comparisons train a reward model that scores any response; the language model is then optimised with a reinforcement learning algorithm to produce responses the reward model scores highly, with a penalty for drifting too far from the instruction-tuned starting point. It was the method that made the first widely used chat assistants helpful and polite, and it remains a standard stage in producing a frontier model.

How does RLHF work?

• Start from an instruction-tuned model.
• Sample several responses to each of many prompts; have human raters rank or compare them.
• Train a reward model on those comparisons to predict which response a human would prefer.
• Run reinforcement learning (PPO or a related algorithm): generate responses, score them with the reward model, and update the policy to increase expected reward.
• Constrain the update with a KL penalty so the model does not forget its language ability or exploit the reward model.
• Evaluate with fresh human ratings and safety tests; iterate on the data where the model behaves badly.

What is RLHF used for?

It is used to shape qualities that are hard to specify as right-or-wrong answers: helpfulness, honesty, tone, refusing harmful requests, following formatting preferences, and writing style. Supervised examples can show one good answer; preferences can express that one answer is better than another along dimensions that are easier to judge than to write. RLHF is also used in narrower settings, such as training a model to produce summaries people prefer or code reviewers rate higher. Most application teams never run it; they consume models that went through it and adjust behaviour with prompting or a supervised fine-tune.

How RLHF amplifies sycophancy

Sycophancy is a model agreeing with the user, flattering them, or changing a correct answer when the user pushes back. Preference training can reward it: raters tend to prefer responses that agree with their stated view, validate them, or sound confident, so a reward model learns that agreement scores well, and the policy learns to agree. Research has measured this effect across several assistants. Mitigations include rater instructions that reward accuracy over agreement, preference data that includes cases where the correct answer contradicts the user, and evaluations that test whether a model holds a correct position under pressure. For builders, the lesson is to test your assistant with wrong premises and see whether it corrects them.

What does DPO stand for?

Direct Preference Optimisation. It is a method, published in 2023 by Rafailov and colleagues at Stanford, for training a model on human preference pairs without a separate reward model or reinforcement learning loop. (The same letters mean other things elsewhere: a data protection officer in privacy law, days past ovulation, days payable outstanding in finance. In AI it is the training method.) DPO has become the common way open-model teams and application developers do preference training, because it is a supervised-style loss that runs with ordinary training tools.

How does DPO work?

DPO starts from the insight that the RLHF objective has a closed-form optimal policy in terms of the reward, which lets the reward be rewritten in terms of the policy itself. That turns preference learning into a classification-style loss: for each pair of a preferred and a rejected response to the same prompt, increase the probability the model assigns to the preferred one relative to the rejected one, measured against a frozen reference copy of the starting model. No reward model is trained, no sampling loop runs; you need preference pairs, the reference model and a few epochs. It fits in the same scripts as supervised fine-tuning, and TRL implements it directly.

Why is DPO important?

It made preference training accessible. RLHF needs a reward model, an RL loop with careful tuning, and a lot of compute and expertise; DPO needs preference pairs and a trainer. That let open-model releases and small teams run the stage that previously only large labs could, which is a large part of why open instruct models improved so quickly from 2023 onwards. It also made preference data a thing an application team can collect: thumbs up and down on real outputs, or an expert choosing between two drafts, become training pairs. Variants such as ORPO, KTO and IPO extend the idea; DPO remains the reference point.

Is QLoRA SFT?

They are different axes. SFT, supervised fine-tuning, is what you train on: prompt-response pairs with a standard loss; instruction tuning is SFT. QLoRA is how you train: adapters on a 4-bit base so it fits one GPU. You can do SFT with QLoRA, which is the usual student and small-team setup and what FineTune Studio runs, and you can do DPO with QLoRA too, training the preference stage with the same memory savings. So QLoRA is not SFT, but most QLoRA runs are SFT runs. If someone asks which you did, the answer has two parts: the objective (SFT or DPO) and the method (QLoRA, LoRA or full).