Frozen Model Training and Testing Architecture

Paper context

Paper title: Generalized Visual Language Models | Lil'Log Abstract: The figure illustrates the architecture of the Frozen model, detailing its training process on the left and its testing pipelines for 0-shot VQA and few-shot classification on the right. Source body: Table of Contents Jointly Training with Image and Text Learned Image Embedding as (Frozen) LM Prefix Text-Image Cross-Attention Fuse Mechanisms No Training Decoding Guided with Vision-based Scores Language as Communication Interface Datasets Image Caption Datasets Pair Image-Text Datasets Evaluation Tasks Visual Question-Answering Visual Language Reasoning Video QA and Understanding Citation References Processing images to generate text, such as image captioning and visual question-answering, has been studied for years. Traditionally such systems rely on an object detection network as a vision encoder to capture visual features and then produce text via a text decoder. Given a large amount of existing literature, in this post, I would like to only focus on one approach for solving vision language tasks, which is to extend pre-trained generalized language models to be capable of consuming visual signals . I roughly group such vision language models (VLMs) into four buckets: Translating images into embedding features that can be jointly trained with token embeddings. Learning good image embeddings that can work as a prefix for a frozen, pre-trained language model. Using a specially designed cross-attention mechanism to fuse visual information into layers of the language model. Combine vision and language models without any training. Jointly Training with Image and Text # One straightforward approach to fuse visual information into language models is to treat images as normal text tokens and train the model on a sequence of joint representations of both text and images. Precisely, images are divided into multiple smaller patches and each patch is treated as one “token” in the input sequence. VisualBERT ( Li et al. 2019 ) feeds both text inputs and image regions into BERT such that it is able to discover the internal alignment between images and text with self-attention mechanism. VisualBERT is trained on the combination of both text and image embeddings. (Image source: Li et al. 2019 ) Similar to text embedding in BERT , each visual embedding in VisualBERT also sums up three types of embeddings, tokenized features $f_o$, segmentation embedding $f_s$ and position embedding $f_p$, precisely: $f_o$ is a visual feature vector computed for a bounding region of the image by a convolutional neural network; $f_s$ is a segment embedding to indicate whether the embedding is for vision not for text; $f_p$ is a position embedding used for aligning the order of bounding regions. The model is trained on MS COCO image caption dataset with both text and image as inputs to predict text captions, using two visually-grounded language model objectives: MLM with the image . The model needs to predict masked text tokens, while image embeddings always stay not masked. Sentence-image prediction . When provided with an image and two associated captions, one of two captions might be a random unrelated caption with 50% probability. The model is asked to distinguish these two situations. According to ablation experiments, the most important configuration is to fuse visual information early on into the transformer layers and to pretrain the model on the COCO caption dataset. Initialization from a pre-trained BERT and the adoption of the sentence-image prediction training objective have relatively small impacts. Ablation study results of VisualBERT on NLVR. (Image source: Li et al. 2019 ) VisualBERT outperforms SoTA at the time on NLVR and Flickr30K, but still has some performance gap with SoTA on VQA. SimVLM (Simple Visual Language Model; Wang et al. 2022 ) is a simple prefix language model , where the prefix sequence is processed with bi-directional attention like BERT, but the main input sequence only has causal attention like GPT . Images are encoded as prefix tokens such that the model can fully consume the visual information and then generates associated text in an autoregressive manner. Inspired by ViT and CoAtNet , SimVLM splits the image into smaller patches in a flatten 1D sequence of patches. They use the convolutional stage consisting of the first 3 blocks of ResNet to extract contextualized patches and this setup is found to work better than a naive linear projection. Training architecture for SimVLM, where the image patches are processed by the cross-attention encoder and the text decoder has causal attention. (Image source: Wang et al. 2022 ) Training data for SimVLM consists of a large number of image-text pairs from ALIGN ( Jia et al. 2021 ) and text-only data from C4 dataset ( Raffel et al. 2019 ). They mix the two pretraining datasets within each batch, containing 4,096 image-text pairs (ALIGN) and 512 text-only documents (C4). According to ablation studies, it is important to have both image-text and text-only data for training. The PrefixLM objective outperforms both span corruption and naive LM. Ablation study results of SimVLM on VQA. (Image source: Wang et al. 2022 ) CM3 (Causally-Masked Multimodal Modeling; Aghajanyan, et al. 2022 ) is a hyper-text language model, learning to generate the content (hypertext markup, hyperlinks and images) of large scale HTML web pages of CC-NEWS and Wikipedia articles. The resulting CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts. Architecture-wise, CM3 is an autoregressive model. However, in order to combine causal and masked language modeling, CM3 also masks out a small number of long token spans and tries to generate them at the end of the sequences. Illustration of how a causally masked language model works. (Image source: Aghajanyan, et al. 2022 ) The training dataset for CM3 contains close to 1T Web data. During preprocessing, images are first downloaded from src and resized to 256 x 256 with random cropping. Then they are tokenized by VQVAE-GAN , resulting in 256 tokens per image. These tokens, joined with spaces, are inserted back into the src attribute. CM3 can be used to c