A vision–language model (VLM) is a type of artificial intelligence system that can jointly interpret and generate information from both images and text, extending the capabilities of large language models (LLMs), which are limited to text. It is an example of multimodal learning. Many widely used commercial applications now rely on this ability. OpenAI introduced computer vision capabilities to its GPT-4V variant of the GPT-4 model, enabling users to incorporate uploaded photographs or diagrams into their discussions with ChatGPT. It has since become an integral part of ChatGPT's standard offering. Similar capabilities were added to Google’s Gemini, Anthropic’s Claude 3 Opus, and Microsoft’s Copilot with Vision. Alongside these models, several open-source vision–language models—such as LLaVA, InstructBLIP, and MiniGPT-4—have been released by the research community, offering smaller-scale alternatives for experimentation and academic study.
History Vision language models evolved from image captioning systems. Such systems were designed to take images alone (without accompanying instructions), and produce descriptions. Most image captioning systems used an encoder-decoder architecture, where an encoder summarized images into feature vectors, which were fed to a decoder to generate the associated description. Early methods (early 2010s), combined handcrafted visual features to encode images, and n-gram or rule-based text templates to generate descriptions. With the rise of deep learning, neural networks became dominant in image captioning. In 2015, methods emerged that used variations of convolutional neural networks (CNN) to encode images, and recurrent neural networks (RNN) to generate the captions. By 2018, transformer networks replaced RNNs in the role of language decoders. Importantly, training of network parameters was based on datasets of image-text pairs, like MS COCO (Common Objects in Context). The scope of applications was also broadened, to include visual question answering (VQA), phrase grounding and others. In 2021, OpenAI's release of CLIP (Contrastive Language–Image Pretraining) was a major step towards the later evolution of VLMs. Rather than focus on a specific task like image captioning, CLIP is a general-purpose foundation model which can be extended to a broad range of downstream tasks. Importantly, CLIP's components were trained on a vast dataset of 400 million image-text pairs, producing powerful models. CLIP's general-purpose structure also places this powerful capability at the disposal of systems with far smaller computational budgets. Starting in 2022, many VLM architectures have been proposed, based on similar design philosophies (elaborated below). These included Google DeepMind's proprietary Flamingo and an open-source variant, LLaVA, SalesForce's InstructBLIP, Microsoft's Kosmos, KAUST's MiniGPT-4 and others. All these merged a separately trained CLIP-like image encoder, an off-the-shelf large language model (LLM) for text encoding, stitched together using specialized components. The resulting joint system was trained on curated datasets. The release of GPT-4V in 2023 marked the emergence of highly-impactful commercial applications. This was quickly followed by other systems mentioned above (including Google’s Gemini, Anthropic’s Claude 3 Opus, and Microsoft’s Copilot with Vision). These applications are substantially more powerful for general-purpose assignments, typically containing substantially more parameters, trained on massive datasets, requiring enormous compute power. Their architectures have not been disclosed.
Architecture The input to VLMs consists of vision elements (images and videos) and text. The output is typically corresponding text. Generative models, which also generate vision elements (e.g., DALL-E), are beyond the scope of this article. Below is a description of a few representative models, for whom the architecture is known. Commercial VLMs like GPT-4V, whose designs were not publicly disclosed, are likely based on similar concepts.
LLaVA 1.0 LLaVA (Large Language and Vision Assistant) 1.0 is a simple model which captures some of the main concepts of open-source VLMs. The input to the model is an image and an accompanying textual language instruction.
Language model backbone Conceptually, the design is built around an off-the-shelf foundation LLM (a fine-tuned variant of Llama called Vicuna), with components patched on to support the image inputs. LLaVA borrows the tokenizer and the transformer modules (including their weights) from Vicuna, and uses them to handle the accompanying text. Recall that in a legacy (non-VLM) application of Vicuna, the tokenizer converts text into a stream of tokens, which are transferred into the transformer module, which in turn produces a stream of response tokens. These are then converted back to text using the tokenizer.
Vision encoding To this, LLaVA adds two components, to support image inputs:
… excerpt ends here. Continue reading the full article.
