Quantizing a fine-tuned model: GGUF, AWQ and GPTQ, and which to pick

A fine-tuned model that doesn't fit your deployment budget needs quantization. The practical differences between the three common formats, in one place.

Written by

Pranjul Rathour

GenAI Engineer at SCULT INDIA, based in Kanpur, India. Builds production RAG systems, fine-tunes open LLMs and ships agentic AI products; has mentored 200+ college developers. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks.

pranjulrathour41@gmail.com LinkedIn GitHub X Instagram Bluesky Dev.to

Presenting BrandHive

A fine-tuned model that scored well in evaluation is useless if it can't run within your deployment budget. Quantization shrinks it — the question is which format.

The three you'll actually encounter

GGUF: for CPU and llama.cpp-style local inference; simplest to deploy, widest tool support, slightly lower throughput on GPU.
AWQ: activation-aware quantization, strong quality retention at 4-bit, good GPU inference throughput.
GPTQ: an earlier and still widely-supported 4-bit method; slightly behind AWQ on quality at the same bit-width in most comparisons.

How to choose

Deploying to a laptop or CPU-only server: GGUF. Deploying to a GPU and chasing throughput: AWQ first, GPTQ if your serving stack doesn't yet support AWQ. Always re-run your evaluation set after quantizing — the quality drop is usually small but not always negligible.

See merging LoRA adapters and serving for the step that usually comes right before this one.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.