The hyperparameter page in most guides lists twenty knobs and explains none. After many runs building and testing FineTune Studio on small models, here is the short list of settings that changed outcomes, with defaults that work for 1B–8B models on a single GPU.

Rank and alpha

Rank (r) sets the capacity of the adapter; alpha scales its contribution. For behaviour and format tasks, r = 8 to 16 is plenty. Raise it to 32–64 for tasks that need more change, such as a new output language or dense domain knowledge. A common, sensible pairing is alpha = 2 × r. Doubling rank doubles adapter size but rarely doubles quality.

Target modules

Attaching adapters only to attention projections is the classic setup. Including the MLP layers as well generally improves results at modest memory cost, and is my default for QLoRA on small models where memory is already handled by quantisation.

Learning rate

The single most sensitive knob. Adapters tolerate higher rates than full fine-tuning; 1e-4 to 2e-4 with a short warmup and cosine or linear decay is a reliable range. If the loss curve drops off a cliff then climbs, the rate is too high. If it barely moves over an epoch, too low.

Epochs versus dataset size

  • A few hundred examples: 3–5 epochs, watch for the eval loss turning upward.
  • A few thousand examples: 1–3 epochs is usually enough.
  • Tens of thousands: one epoch, and consider whether you need all of it.

Batch size and gradient accumulation

On a small GPU, per-device batch of 1–4 with gradient accumulation to an effective batch of 16–32 is the normal pattern. Accumulation trades time for memory; it does not change the maths. Sequence length is the hidden memory hog: cap it at what your data needs, not at the model's maximum.

Settings people over-tune

Dropout on adapters, weight decay, scheduler exotica. They matter at the margins. Reading the loss curve — see how to read a loss curve during fine-tuning — matters far more than any of them.

Start with the defaults above, change one thing at a time, and let a held-out evaluation, not the training loss, tell you what improved.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.