Models that read pictures as well as words.

What is vision language model?

A vision-language model is a neural network trained on images paired with text so that it can relate the two. Depending on its design it can embed an image and a sentence into the same vector space (for matching and search), generate a caption or an answer from an image (for description and question answering), or ground text in image regions (for pointing to where something is). The term covers both the contrastive models like CLIP and the generative multimodal language models; the common thread is training on millions of image-text pairs so that visual and linguistic concepts line up.

How do vision language models work?

There are two main recipes. Contrastive: an image encoder and a text encoder are trained together so that matching image-caption pairs end up close in a shared space and mismatched pairs far apart; the result is a model that scores how well any text describes any image. Generative: a vision encoder turns the image into a sequence of visual tokens, a small projection maps them into the language model's embedding space, and the language model then treats them like words, so it can answer questions about the image by generating text. Most current multimodal LLMs use the second recipe, often with a contrastively pretrained encoder as the vision part.

What is CLIP model?

CLIP (Contrastive Language-Image Pretraining) is a model released by OpenAI in 2021 that was trained on hundreds of millions of image-caption pairs from the web with the contrastive recipe. It has an image encoder and a text encoder; given an image and several candidate texts it returns which text fits best, without any task-specific training, which is called zero-shot classification. Its embeddings are also used directly for image search, deduplication and as the vision backbone inside many multimodal LLMs. Open re-creations with public data and improved training exist, and the name CLIP is often used for the whole family.

What does CLIP model do?

• Zero-shot classification: label an image by comparing it with text prompts such as 'a photo of a cat' and 'a photo of a dog'.
• Text-to-image search: embed a query sentence and find the nearest image embeddings in a collection.
• Image-to-image search and deduplication by embedding distance.
• Content filtering and tagging at scale without training a classifier.
• Guiding image generation models and scoring how well a generated image matches a prompt.
• Serving as the vision encoder that feeds visual tokens into multimodal language models.

What is multimodal LLM?

A multimodal LLM is a language model that accepts more than text as input, images most commonly, and increasingly audio, video and documents, and produces text (and in some cases images or speech) as output. You can paste a photo of a whiteboard and ask for the equations, send a chart and ask what it shows, give it a scanned invoice and ask for the fields as JSON, or show it a screenshot and ask what to click. The frontier hosted models are multimodal, and open models from the Qwen, Llama, Gemma and other families have vision variants that run locally through Ollama.

How do multimodal LLMs work?

The image is split into patches and encoded by a vision transformer into a grid of feature vectors. A projector (a small network) maps those vectors into the same dimensionality as the language model's token embeddings, producing a few hundred to a few thousand visual tokens per image. Those tokens are placed in the prompt sequence where the image appears, and the language model attends across both visual and text tokens to generate its answer. Training proceeds in stages: align the projector on captions, then instruction-tune on image-question-answer data. Higher-resolution handling, multiple images and video are extensions of the same scheme with more tokens.

VLM vs multimodal LLM

The terms overlap. VLM is the broader category: any model connecting vision and language, including contrastive models like CLIP that do not generate text, captioners, grounding models and generative question-answering models. Multimodal LLM refers specifically to a large language model extended to accept images (and possibly other modalities) in its prompt. So every multimodal LLM is a VLM, but CLIP is a VLM that is not an LLM. In practice, when someone says VLM today they usually mean a generative model you can chat with about images; when they say CLIP-style they mean embeddings for matching.

What is vision language action model?

A vision-language-action (VLA) model extends a VLM to output actions rather than only text, for controlling robots. It takes camera images and a language instruction ('put the cup in the sink'), and produces a sequence of motor commands or waypoints, having been trained on demonstrations where images, instructions and actions were recorded together. The language model's world knowledge and the vision encoder's perception transfer to new objects and instructions better than earlier robot policies. It is an active research area with early open models and datasets; for most students it is a topic to follow rather than a tool to use yet.

What is the best vision language model?

For hosted use, the current frontier multimodal models from the major providers lead on hard visual reasoning and document understanding, and their rankings change with each release. For local use, open vision models from the Qwen, Llama, Gemma, InternVL and Pixtral families cover most needs, with small variants that run on a laptop through Ollama. For embeddings and search, CLIP-style models and their open successors (SigLIP and similar) are the standard. As with every model choice on this site: shortlist three, run thirty of your own images and questions, score blind, and check cost and latency. Document extraction in particular should be validated field by field, as DocuLens AI does, whichever model wins.

Vision language model tutorial

• Zero-shot classification with CLIP: load an open CLIP model with Transformers or open_clip, encode an image and a few text labels, pick the highest similarity. Ten lines.
• Image search: embed a folder of photos with the same model, store vectors in FAISS, embed a text query, return the nearest images.
• Chat with an image locally: pull a vision model in Ollama and send a photo with a question through the API; ask for JSON when you want fields.
• Document extraction: send a scanned invoice to a multimodal model with a schema, parse the JSON, validate every field against type and format rules, and show the page region for each.
• Measure: build thirty labelled images and questions; compare two models on accuracy, latency and cost before you commit.