The Vision Language Model (VLM) is the most popular type of multimodal AI model in the past two years: it can read text and "see" images, and based on this, answer questions, describe scenes, read charts, and even complete tasks that require joint reasoning between text and images. GPT-4o, Gemini, Claude, and Qwen VL all belong to this family. This article uses as few formulas as possible to explain the basic structure, training process, and common capability boundaries of VLM.
1、 The basic structure of VLM: three-stage structure
A typical VLM can be roughly divided into three parts:
1. Vision Encoder: Cut the image into small squares (patches) and encode them into a string of "visual features". The mainstream approach is ViT (Vision Transformer), which no longer outputs pixels, but a set of vectors.
2. Projector/Adapter: Translate visual features into a vector space that language models can understand. The common practice is a simple MLP projection layer (such as LLaVA), while others use Q-Former (such as BLIP-2) or cross attention.
3. Language Model (LLM) backbone: Receive the concatenated sequence of "visual token+text token" and generate answers through autoregression like processing regular text.
So from a model perspective, the essence of "looking at pictures" is to turn the pictures into a special string of tokens and stuff them into the context of the language model.
2、 Training usually involves several steps
The training approach for mainstream open-source VLMs (such as the LLaVA series) is roughly as follows:
-Stage 1: Align pre training. Freeze the visual encoder and LLM, train only the projection layer, and align visual features to the language space using text and images.
-Stage 2: Instruction fine-tuning. Unfreeze LLM (or all parameters), train the model with graphic and textual instruction data, and let the model learn to describe, answer questions, and reason according to human instructions.
-Some of the solutions will also include higher resolution block encoding, stronger OCR and document understanding data, and domain specific fine-tuning for medical and industrial quality inspection.
3、 Several easily confused points
-CLIP is not VLM. CLIP performs image text matching (contrastive learning), which can determine whether the image and text are similar, but does not generate sentences. On the basis of CLIP visual encoder, VLM adds a language model that can be generated.
-'Visible' does not mean 'understandable'. High resolution small characters, dense charts, and scenes that require counting are still the weaknesses of VLM.
-Illusions also exist. VLM also tends to use "picture editing", especially when the image is blurry or the problem exceeds the content of the image, it is easy to give seemingly reasonable but incorrect descriptions.
4、 Typical applications
-Understanding documents and receipts: from scanned copies PDF、 Extract structured information from the table;
-Accessibility assistance: generate descriptions for images;
-E-commerce and Content Review: Identify products, scenarios, and illegal content;
-Industrial and Medical: Combining professional data for defect detection and initial screening of images (such scenarios often require additional domain fine-tuning and rigorous validation).
5、 Practical advice for developers
-Firstly, clarify the task: whether it is "description," "extraction," or "inference," as the requirements for model size and resolution vary greatly among different tasks;
-Resolution and segmentation strategies are often more effective than switching to a larger model;
-For outputs related to professional fields, the manual review process must be retained;
-Evaluation should use one's own real data, and public rankings can only be used for rough screening.
[Reference source] Comprehensive compilation of industry information and mainstream open source project documents (including CLIP, BLIP-2, LLaVA and other public papers and project descriptions).