vLLM: one GPU, many users.

What is vLLM?

vLLM is an open-source library and server, started at UC Berkeley in 2023, for fast inference of large language models. You point it at a model from the Hugging Face Hub and it serves an OpenAI-compatible HTTP API, so existing client code works unchanged. Its purpose is efficiency when many requests arrive at once: it schedules them onto the GPU together, shares memory cleverly, and streams each response back. It supports most open model architectures, tensor parallelism across several GPUs, quantized weights (AWQ, GPTQ, FP8 and others), LoRA adapters served alongside a base model, and speculative decoding.

How does vLLM work?

Two ideas carry most of the gain. Paged attention stores the key-value cache (the memory of a request's tokens that attention reads) in fixed-size blocks allocated on demand, instead of reserving one large contiguous region per request. That removes fragmentation and lets memory be shared between requests that have the same prefix, so far more requests fit on a GPU. Continuous batching adds and removes requests from the running batch at every decoding step, so a short answer finishing does not make others wait and a new request starts immediately. Together they turn one GPU from serving a few users into serving dozens with similar latency.

How does vLLM handle concurrent requests?

A scheduler maintains a queue of waiting requests and a running batch. At each step it decides which requests to run together, based on available key-value cache blocks, and executes one forward pass for all of them, generating one token per request. Requests that finish free their blocks; waiting requests are admitted as blocks become available; if memory is exhausted, some requests are pre-empted and resumed later. Prefix caching lets requests with a shared system prompt reuse the same cached blocks. From the client's side it looks like independent streaming responses; underneath it is one tightly packed loop.

Why is vLLM faster than Ollama?

They are optimised for different situations. Ollama, built on llama.cpp, is tuned for a single user on a laptop: quick model switching, CPU and Apple Silicon support, GGUF files, minimal setup. Under one request at a time the two are comparable. vLLM is tuned for many simultaneous requests on a data-centre or workstation GPU: batching, paged memory and optimised CUDA kernels let it produce far more total tokens per second when ten or a hundred users are active. So vLLM is faster where it matters for serving an application, and Ollama is simpler where it matters for a person at a desk.

vLLM vs Ollama

• Audience: Ollama for individuals and prototypes; vLLM for applications serving many users.
• Hardware: Ollama runs on CPU, Apple Silicon and GPUs; vLLM needs a supported GPU and Linux.
• Formats: Ollama uses GGUF; vLLM uses Hub checkpoints in 16-bit or AWQ, GPTQ, FP8 (GGUF support is experimental).
• Throughput: comparable for one user; vLLM far ahead under concurrency.
• Setup: Ollama is one install and one command; vLLM is a Python package or Docker image with GPU drivers to match.
• API: both expose OpenAI-compatible endpoints, so the application code can be the same.

vLLM vs Hugging Face Transformers

Transformers is the reference implementation: flexible, used for training and research, correct but not optimised for serving. Calling generate on a Transformers model handles one batch at a time with a contiguous key-value cache and Python overhead per token. vLLM reimplements the inference path with custom kernels, paged memory and a scheduler, and typically produces many times the throughput under load from the same weights. Use Transformers to fine-tune, experiment and evaluate; use vLLM (or an equivalent such as SGLang or TGI) to serve the result. vLLM loads the same Hub checkpoints Transformers does.

How to install vLLM

On Linux with an NVIDIA GPU: create a virtual environment with a supported Python version, install vLLM with pip (it brings a matching PyTorch and CUDA build), and start the server with a model name from the Hub. The official Docker image avoids driver and version mismatches and is what I use on servers. AMD GPUs have their own build. Check that the GPU has enough memory for the model plus key-value cache, and set the maximum model length and memory utilisation flags to fit. Test with a curl to the chat completions endpoint before pointing an application at it.

Can vLLM run on Windows?

Not natively. vLLM targets Linux; on a Windows machine with an NVIDIA GPU the supported route is WSL2 (Windows Subsystem for Linux) with the CUDA driver that exposes the GPU to WSL, then installing vLLM inside the Linux environment, or running the Docker image with GPU passthrough through WSL2. Both work well in practice. Community Windows ports exist but lag behind and are not supported. If you only need a local model on Windows for yourself, Ollama runs natively and is the easier choice; vLLM is worth the WSL2 setup when you are serving an application.

Can vLLM run on CPU?

There is a CPU backend for x86 processors with AVX-512 and for ARM, intended for development and for environments without a GPU, but it is not what vLLM is for and throughput is far below GPU serving. For CPU inference, llama.cpp and Ollama with GGUF models are better engineered and much faster, because their quantization formats and kernels were designed for CPUs. Use vLLM's CPU mode to test a pipeline on a laptop before deploying to a GPU server; use llama.cpp or Ollama when CPU is the production target.

Does vLLM support GGUF?

Experimentally. vLLM can load some GGUF-quantized models, with limits on which architectures and quantization levels work and without the performance of its native quantization formats. If you are coming from Ollama with a GGUF file, the better route for vLLM is to use the original Hub checkpoint and either serve it in 16-bit or use an AWQ or GPTQ version, which have optimised kernels. Keep GGUF for llama.cpp and Ollama, where it is first-class, and use vLLM's own formats on the GPU server; the model is the same, only the container differs.