Courses·Advanced And Emerging·8 min read

Vision-Language Models (VLMs)

Vision-Language Models integrate visual perception with language understanding in a single system, enabling AI to see, reason about, and describe the visual world -- and increasingly, to act on it through Vision-Language-Action architectures.

One-Line Summary: Vision-Language Models integrate visual perception with language understanding in a single system, enabling AI to see, reason about, and describe the visual world -- and increasingly, to act on it through Vision-Language-Action architectures.

Prerequisites: Understanding of Transformer architecture and attention mechanisms, familiarity with image encoders (CNNs, Vision Transformers), text embeddings and language model generation, basic awareness of multimodal learning and cross-modal alignment.

What Are Vision-Language Models?

Imagine a person who can read but has never seen an image, and another person who can see but has never read. Neither can fully understand a newspaper: the reader misses the photographs, the viewer misses the articles. A VLM is like a person who can do both -- look at an image and discuss what they see in natural language, answer questions about visual content, or follow instructions that require understanding both text and images.

flowchart TD
    L1["vision encoder"]
    L2["projection layer"]
    L3["language model backbone"]
    L1 --> L2
    L2 --> L3

VLMs combine two fundamentally different types of information processing. Vision involves spatial relationships, textures, colors, object recognition, and scene understanding. Language involves sequential reasoning, abstraction, and symbolic manipulation. The challenge is not just having both capabilities but deeply integrating them so the model can reason about the relationship between what it sees and what it reads.

This is not simply "image captioning plus a chatbot." Modern VLMs can understand charts and graphs, read handwritten text, analyze medical images, navigate user interfaces, interpret diagrams, and follow visual instructions -- tasks that require genuine cross-modal reasoning.

How It Works

flowchart LR
    S1["CLIP contrastive learning"]
    S2["image-text pairs aligned in shared embeddi"]
    S1 --> S2

The Three-Component Architecture

Almost all VLMs share a common architectural pattern with three components:

1. Vision Encoder

Converts raw images into a sequence of visual embeddings (analogous to how a tokenizer converts text into token embeddings). The most common encoders are:

Vision Transformers (ViT): Split the image into fixed-size patches (typically 14x14 or 16x16 pixels), project each patch into an embedding, and process the sequence through Transformer layers. This produces a sequence of visual tokens.
SigLIP / CLIP encoders: Vision encoders pre-trained with contrastive learning to align visual representations with text. These encoders have already learned to represent images in a way that is meaningful to language.

A 224x224 image with 14x14 patches produces 256 visual tokens. Higher-resolution images or smaller patches produce more tokens (a 448x448 image with 14x14 patches produces 1,024 visual tokens), trading compute for visual detail.

2. Cross-Modal Connector (Projection Module)

Bridges the vision encoder's output space with the language model's input space. Since the vision encoder and language model were typically trained independently, their embedding spaces do not naturally align. The connector projects visual embeddings into the language model's embedding space. Common approaches:

Linear projection: A simple learned linear layer (as in LLaVA). Surprisingly effective.
MLP projection: A multi-layer perceptron with nonlinearities for more expressive mapping.
Q-Former (as in BLIP-2): A small Transformer that uses learnable query tokens to extract a fixed number of visual features from the vision encoder, compressing visual information.
Perceiver Resampler (as in Flamingo): Uses cross-attention to compress a variable number of visual tokens into a fixed set of latent tokens.

3. Language Model (LLM backbone)

The autoregressive language model that receives the projected visual tokens interleaved with text tokens and generates the text response. The visual tokens are typically prepended to (or interleaved with) the text tokens in the input sequence, and the language model attends to both modalities through its standard attention mechanism.

Training Pipeline

VLM training typically proceeds in stages:

Stage 1: Pre-training alignment (vision-language connection)

The vision encoder and language model are frozen. Only the cross-modal connector is trained on large-scale image-caption pairs (hundreds of millions of pairs). The goal is to learn the mapping between visual and textual representations. This is relatively cheap because only the small connector module is trained.

Stage 2: Visual instruction tuning

The language model (and sometimes the vision encoder) is unfrozen and fine-tuned on curated visual instruction-following data -- tasks like visual question answering, image description, chart interpretation, document understanding, and multi-turn visual dialogue. This teaches the model to use its visual understanding in practical ways.

Stage 3: Alignment and safety training

RLHF or DPO applied to visual tasks, teaching the model to refuse harmful visual requests, avoid hallucinating visual details, and follow instructions accurately.

Key Model Families

GPT-4V / GPT-4o (OpenAI): Proprietary multimodal model with strong visual reasoning, chart understanding, and document analysis.
Claude 3/3.5/4 (Anthropic): Integrated vision capabilities with strong document understanding and visual reasoning.
Gemini (Google): Natively multimodal architecture designed from the ground up for cross-modal reasoning.
LLaVA (open-source): The most influential open VLM architecture, demonstrating that simple linear projection + visual instruction tuning can achieve strong results.
Qwen-VL (Alibaba): Strong open VLM with high-resolution image understanding and multilingual capabilities.

Vision-Language-Action (VLA) Models

VLA models extend VLMs by adding motor control output, creating systems that can see, reason, and physically act. The model receives visual input (camera feed), processes it through the VLM pipeline, and outputs motor actions (joint angles, gripper commands, navigation waypoints).

This represents a new frontier where AI bridges perception, language understanding, and physical interaction:

RT-2 (Google): A VLA model that translates visual observations and language instructions into robot actions.
Octo, OpenVLA: Open-source VLA models for robotic manipulation.

Hierarchical and late-fusion architectures -- where high-level reasoning and low-level motor control are handled by separate but connected modules -- have been shown to achieve the highest success rates for manipulation tasks and generalization to new objects and environments.

Why It Matters

VLMs represent a qualitative expansion of what AI systems can do:

Document intelligence: Analyzing PDFs, invoices, contracts, scientific papers, and forms that combine text, tables, charts, and images. This is one of the highest-value enterprise applications.
Accessibility: Describing images, reading visual content, and interpreting visual information for visually impaired users.
UI automation: Understanding and interacting with graphical user interfaces, enabling AI agents to operate software by "seeing" the screen.
Medical imaging: Interpreting X-rays, MRIs, pathology slides, and other medical images with natural language explanations, potentially democratizing expert-level diagnostic capability.
Autonomous systems: Self-driving cars, drones, and robots that need to understand their visual environment and act on natural language instructions.
Scientific discovery: Analyzing microscopy images, satellite imagery, astronomical data, and other visual scientific data with language-based reasoning.

Key Technical Details

Resolution vs. token count trade-off: Higher image resolution provides more visual detail but produces more visual tokens, which consume context window capacity and increase compute cost. Dynamic resolution strategies (using high resolution only for relevant image regions) are an active area of optimization.
Hallucination in VLMs: VLMs suffer from a specific form of hallucination: confidently describing visual details that are not present in the image. This is particularly concerning for safety-critical applications (medical imaging, autonomous driving).
OCR capability: Modern VLMs can read text in images (OCR) with high accuracy, enabling document understanding without separate OCR pipelines. However, performance varies significantly with text size, font, orientation, and image quality.
Multi-image understanding: Processing multiple images in a single query (for comparison, sequence understanding, or multi-view reasoning) is an emerging capability with significant practical value.
Video understanding: Extending VLMs to video by sampling frames and processing them as multiple images. The challenge is efficiently representing temporal information without exhausting the context window.
Visual grounding: The ability to not just describe what is in an image but point to specific regions (bounding boxes, segmentation masks) in response to natural language queries.

Common Misconceptions

"VLMs understand images the way humans do." VLMs process images as sequences of patch embeddings and learn statistical associations between visual patterns and text. They can be fooled by adversarial patches, struggle with spatial reasoning, and may hallucinate plausible but incorrect visual descriptions.
"Adding vision to an LLM is straightforward." The cross-modal alignment problem is non-trivial. Naive approaches produce models that can describe images but cannot reason about them. Effective VLMs require carefully staged training and high-quality visual instruction data.
"VLMs replace specialized computer vision models." For well-defined tasks (object detection, semantic segmentation, pose estimation), specialized models are often more accurate and efficient. VLMs excel at open-ended visual reasoning and flexible instruction following.
"Higher resolution always means better performance." Beyond a certain point, increased resolution adds compute cost without proportional quality improvement, especially for tasks that do not require fine visual detail.
"VLMs can reliably count objects in images." Counting is a known weakness. VLMs often struggle to accurately count objects, especially when there are many similar items.

Connections to Other Concepts

multimodal-models.md: VLMs are the most mature subclass of multimodal models. The principles of cross-modal alignment, multi-stage training, and modality-specific encoders generalize to audio, video, and other modalities.
transformer-architecture.md: VLMs rely on the Transformer's ability to attend across modalities when visual and text tokens are combined in the input sequence.
token-embeddings.md: Visual tokens are projected into the same embedding space as text tokens, leveraging the language model's existing representational capacity.
flash-attention.md: Essential for processing the large token counts that arise from high-resolution images combined with text.
ai-agents.md: VLMs enable visual grounding for agents -- understanding screenshots, reading web pages, interpreting dashboards, and navigating graphical interfaces.
prompt-injection.md: VLMs introduce new attack surfaces: adversarial images can contain hidden text or patterns that manipulate model behavior.