Quantization: the same model in fewer bits.

What is quantization in LLM?

A language model is a few billion numbers, its weights, normally stored as 16-bit floating point. Quantization maps those numbers to a smaller set of values that fit in fewer bits, typically 8 or 4, with scaling factors that let the original range be approximately recovered. An eight-billion-parameter model is about sixteen gigabytes at 16-bit and about four to five at 4-bit. The model's behaviour is almost the same because the weights carry redundancy; small rounding errors average out across millions of parameters. Quantization is why large models run on laptops at all.

How quantization works in LLM

Weights are grouped into blocks (often 32 to 128 values). For each block, a scale and sometimes an offset are computed so that the block's values map onto the integer range of the target bit width; each weight is rounded to its nearest representable value and stored as a small integer, with the scale kept in higher precision. At inference the integers are dequantized on the fly, or the matrix multiplication runs in integer arithmetic directly. Better methods choose the rounding to minimise the error on real activations rather than on weights alone, which is what distinguishes GPTQ and AWQ from naive rounding, and keep the most sensitive weights in higher precision.

Does quantization reduce accuracy?

A little, and the amount depends on the bit width, the method and the model. 8-bit quantization is close to lossless on standard benchmarks. Good 4-bit quantization typically costs a fraction of a point to a couple of points on benchmarks and is hard to notice in ordinary use, which is why 4-bit is the default on consumer hardware. Below 4 bits (3-bit, 2-bit) the degradation grows quickly: more mistakes, weaker reasoning, more repetition. Smaller models suffer more from quantization than larger ones, so a 4-bit thirty-billion-parameter model often beats a 16-bit eight-billion one in the same memory. Always test on your own prompts.

Does quantization speed up inference?

Usually, for two reasons. Generation is limited by memory bandwidth: every token requires reading all the weights, so weights that are a quarter the size are read four times faster. And smaller weights fit in faster memory, on a GPU rather than spilling to system RAM, or in cache rather than main memory. On GPUs, methods with optimised kernels (AWQ, GPTQ, Marlin) deliver the speed-up; a format without fast kernels can be slower than 16-bit despite being smaller. On CPUs, GGUF with llama.cpp is designed for exactly this and is the main way to get usable speed without a GPU.

How does quantization affect LLM performance?

Three things move. Memory falls, so a bigger model fits or more requests fit at once. Speed rises when the runtime has kernels for the format. Accuracy falls slightly, concentrated in tasks that need precise recall or long chains of reasoning. In practice the trade I recommend: on a laptop, run the largest model that fits at 4-bit rather than a smaller model at higher precision; on a server, use 8-bit or a well-made 4-bit (AWQ or GPTQ) for throughput; for fine-tuning, use QLoRA's 4-bit base with adapters trained in higher precision. And measure with the same thirty prompts before and after quantizing.

When does quantization take place?

After training, in most cases: a model is trained in 16-bit, then quantized once (post-training quantization) and distributed in that form, which is what the GGUF, AWQ and GPTQ files on the Hub are. Some methods calibrate on a small sample of text during that step to choose better roundings. Quantization-aware training, where the model is trained knowing it will be quantized, gives better low-bit results but needs the training run, so it is used by model makers rather than users. In QLoRA the base is quantized at load time for training while the adapters stay in higher precision; the result is merged back at 16-bit or re-quantized for deployment.

What are quantization levels?

Levels are the bit widths and schemes a model is offered in. In GGUF files you see names such as Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_S and Q2_K: the number is the bits per weight, K marks the newer block-wise scheme, and S, M, L mark small, medium and large variants that keep more or fewer layers in higher precision. Q4_K_M is the common sweet spot: about a quarter of the 16-bit size with little visible loss. Q8_0 is near-lossless and twice the size. Q2 and Q3 are for fitting a model that otherwise would not run at all, with noticeable quality loss.

What is GGUF?

GGUF is the file format used by llama.cpp and the tools built on it, including Ollama and LM Studio, to store a model's weights (usually quantized), its tokenizer and its metadata in a single file. It replaced the older GGML format in 2023 and is designed for fast loading with memory mapping and for running on CPUs, Apple Silicon and GPUs through one runtime. Most popular open models are published on the Hub in GGUF at several quantization levels by the community, so you can download the one that fits your memory and run it immediately.

What is GGUF quantization?

It is the family of block-wise quantization schemes that GGUF files use, implemented in llama.cpp. Weights are stored in blocks with per-block scales, in legacy (Q4_0, Q8_0) or K-quant variants (Q4_K_M and friends) that mix precisions across layers to protect sensitive ones, plus importance-matrix variants that calibrate on sample text to reduce error further. You produce them with llama.cpp's conversion and quantize tools from a 16-bit Transformers checkpoint, which is also how a fine-tuned model from FineTune Studio ends up running in Ollama: merge the adapter, convert to GGUF, quantize to Q4_K_M or Q8_0.

What are GGUF files?

GGUF files are the downloadable artefacts: one file per model and quantization level, named with the model, size and level, from under a gigabyte for tiny models to tens of gigabytes for large ones. Each is self-contained; point llama.cpp, Ollama (through a Modelfile) or LM Studio at it and it runs. The Hub hosts them under community accounts that publish every popular model in every level shortly after release. When choosing, pick the largest level your memory holds with headroom for the context window, and prefer K-quant M variants unless you have a reason not to.