LoRA: fine-tune the adapter, not the model.

What is LoRA in AI?

In AI, LoRA is a technique for adapting a large pretrained model to a new task by training a small number of added parameters rather than all of the original ones. It applies to language models, image generators and vision models alike; in image generation a LoRA is often a style or character adapter you download and apply. (Unrelated: LoRa, with a lowercase a, is a long-range radio protocol for sensors; the search results mix them up.) For language models, LoRA is the default way fine-tuning is done today because it is cheap, fast and produces a compact, swappable result.

What does LoRA stand for?

Low-Rank Adaptation. The name comes from the maths: the change to a weight matrix during fine-tuning is approximated as the product of two much smaller matrices, which has low rank. If a layer's weight matrix is 4,096 by 4,096, a rank-8 LoRA trains a 4,096 by 8 matrix and an 8 by 4,096 matrix instead, about one two-hundred-and-fiftieth of the parameters, and the observation from the 2021 paper by Hu and colleagues at Microsoft was that this tiny update captures most of what full fine-tuning achieves.

How LoRA works LLM

During training, the base model's weights are frozen; no gradients are stored for them. For chosen layers, usually the attention projections and sometimes the feed-forward layers, LoRA adds a parallel path: input times matrix A times matrix B, scaled by a factor, added to the frozen layer's output. Only A and B receive gradients, so optimiser memory shrinks to a fraction. At inference the adapter can be merged into the base weights (one addition per layer, no extra latency) or kept separate so several adapters share one base model. The rank, the scaling factor and which layers get adapters are the main settings.

What are LoRA adapters?

An adapter is the trained output of a LoRA run: the A and B matrices for every adapted layer, saved as a small file alongside a configuration that records the rank, the target layers and the base model. Adapters are typically a few megabytes to a few hundred megabytes, against many gigabytes for a full model. Because the base is unchanged, you can keep one base loaded and swap adapters per task or per customer, serve several adapters at once, and share an adapter publicly without redistributing the base. Hugging Face hosts thousands of them for popular base models.

What is QLoRA?

QLoRA (quantised LoRA) is LoRA with the frozen base model stored in 4-bit precision instead of 16-bit, using a quantisation format designed to keep accuracy (NF4), with the computation for each layer dequantised on the fly and the adapters trained in higher precision. Introduced in 2023 by Dettmers and colleagues, it cut the memory needed to fine-tune a large model by roughly four times, which is what made fine-tuning on a single consumer GPU practical. Quality is close to 16-bit LoRA on most tasks. FineTune Studio uses QLoRA for exactly this reason: it is what fits the hardware students and small teams actually have.

What is LoRA and QLoRA?

They are the same training method with different memory footprints. LoRA keeps the base model in its normal 16-bit precision and trains adapters; it needs a GPU that can hold the full model plus activations, which for an eight-billion-parameter model means roughly sixteen gigabytes and more. QLoRA compresses the base to 4-bit and trains the same adapters; the same model then fits in well under ten gigabytes. If you have the memory, LoRA is slightly faster and marginally more accurate. If you do not, QLoRA gets you nearly the same result on hardware you own. The adapter you produce is usable either way.

What is PEFT and LoRA?

PEFT, parameter-efficient fine-tuning, is the family of methods that adapt a model by training only a small subset of parameters. LoRA is the most used member; others include prefix tuning, prompt tuning, adapters inserted between layers, and IA3. PEFT is also the name of the Hugging Face library that implements these methods on top of Transformers: you load a base model, wrap it with a LoRA configuration, and train with the usual tools. When people say PEFT in practice they usually mean LoRA via the PEFT library, and QLoRA is PEFT's LoRA combined with 4-bit loading.

How to fine tune LLM with LoRA

• Load a base model with Transformers, in 4-bit if using QLoRA.
• Define a LoRA configuration: rank (8 to 64 is typical), scaling, dropout, and the target modules (attention projections, optionally feed-forward).
• Wrap the model with PEFT; confirm the trainable parameter count is a small fraction of the total.
• Format your dataset in the model's chat template; tokenise; hold out a validation split.
• Train with TRL's supervised fine-tuning trainer for one to three epochs with a learning rate around one to two times ten to the minus four; watch validation loss.
• Save the adapter; evaluate base versus tuned on held-out prompts; merge or export to GGUF for Ollama if it wins.

LoRA vs fine tuning LLM

LoRA is a way of fine-tuning, so the real comparison is LoRA against full fine-tuning. Full fine-tuning updates every weight: maximum flexibility, highest memory (several times the model size for gradients and optimiser state), a full-size output per task, and a greater risk of forgetting general abilities. LoRA updates a small adapter: a fraction of the memory, hours instead of days, a small output, less forgetting, and in most reported comparisons on instruction and task data, nearly the same quality. For almost everyone outside a research lab, LoRA or QLoRA is the fine-tuning method.

What does LoRA accomplish in fine tuning?

It makes fine-tuning affordable, reversible and composable. Affordable because memory and time drop to what a single GPU can supply. Reversible because the base is untouched; remove the adapter and you have the original model. Composable because adapters can be swapped, stacked for some use cases, and served side by side on one base. In practice LoRA turned fine-tuning from a research-lab activity into something a student can do on a weekend and a small team can run for every customer. The quality of the result still comes from the dataset; LoRA only removes the hardware excuse.