Unsupervised Data Generation (UDG) Framework Overview

Paper context

Paper title: Learning with not Enough Data Part 3: Data Generation | Lil'Log Abstract: The figure illustrates the Unsupervised Data Generation (UDG) framework, showing how a GLM generates synthetic data from few-shot examples to fine-tune a downstream BERT/T5 model. Source body: Table of Contents Data Augmentation Image Augmentation Basic Image Processing Operations Task-Specific Augmentation Strategies Image Mixture Text Augmentation Lexical Edits Back-translation Mix-up Audio Augmentation Architectural Augmentation Data Synthesis Language Model as Noisy Annotator Language Model as Data Generator How to Quantify Generated Data Quality? Training with Noisy Data Regularization and Robust Architecture Robust Learning Objective Label Correction Sample Reweighting and Selection Citation Reference Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2 ). Let’s consider two approaches for generating synthetic data for training. Augmented data . Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits. New data . Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM) . Few shot prompting is shown to be effective for LM to learn within context without extra training. Data Augmentation # The goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged. Image Augmentation # Basic Image Processing Operations # There are several ways to modify an image while retaining its semantic information. We can use any one of the following augmentation or a composition of multiple operations. Random cropping and then resize back to the original size. Random color distortions Random Gaussian blur Random color jittering Random horizontal flip Random grayscale conversion And many more. Check PIL.ImageOps for inspiration. Task-Specific Augmentation Strategies # If the downstream task is known, it is possible to learn the optimal augmentation strategies (i.e. what processing operations to use and how to combine them in sequence) to maximize the downstream task performance. AutoAugment ( Cubuk, et al. 2018 ) is inspired by neural architecture search , AutoAugment frames the problem of learning best data augmentation operations (i.e. shearing, rotation, invert, etc.) for image classification as an RL problem and looks for the combination that leads to the highest accuracy on the evaluation set. AutoAugment can be executed in adversarial fashion ( Zhang, et al 2019 ). RandAugment ( Cubuk et al., 2019 ) greatly reduces the search space of AutoAugment by controlling the magnitudes of different transformation operations with a single magnitude parameter. Population based augmentation (PBA; Ho et al., 2019 ) combines PBT (“population based training”; Jaderberg et al, 2017 ) with AutoAugment, using the evolutionary algorithm to train a population of children models in parallel to evolve the best augmentation strategies. Unsupervised Data Augmentation (UDA; Xie et al., 2019 ), among a set of possible augmentation strategies, selects a subset to minimize the KL divergence between the predicted distribution over an unlabelled example and its unlabelled augmented version. Image Mixture # Image mixture methods can construct new training examples from existing data points. Mixup ( Zhang et al., 2018 ) runs global-level mixture by creating a weighted pixel-wise combination of two existing images $I_1$ and $I_2$: $I_\text{mixup} \gets \alpha I_1 + (1-\alpha) I_2$ and $\alpha \in [0, 1]$. Cutmix ( Yun et al., 2019 ) does region-level mixture by generating a new example by combining a local region of one image with the rest of the other image. $I_\text{cutmix} \gets \mathbf{M}_b \odot I_1 + (1-\mathbf{M}_b) \odot I_2$, where $\mathbf{M}_b \in \{0, 1\}^I$ is a binary mask and $\odot$ is element-wise multiplication. It is equivalent to filling the cutout ( DeVries & Taylor 2017 ) region with the same region from another image. Given a query $\mathbf{q}$, MoCHi (“mixing of contrastive hard negatives”; Kalantidis et al. 2020 ) maintains a queue of $K$ negative features $Q={\mathbf{n}_1, \dots, \mathbf{n}_K }$ and sorts these negative features by similarity to the query, $\mathbf{q}^\top \mathbf{n}$, in descending order. The first $N$ items in the queue are considered as the hardest negatives, $Q^N$. Then synthetic hard examples can be generated by $\mathbf{h} = \tilde{\mathbf{h}} / |\tilde{\mathbf{h}}|_2$ where $\tilde{\mathbf{h}} = \alpha\mathbf{n}_i + (1-\alpha) \mathbf{n}_j$ and $\alpha \in (0, 1)$. Even harder examples can be created by mixing with the query feature, $\mathbf{h}’ = \tilde{\mathbf{h}’} / |\tilde{\mathbf{h}’}|_2$ where $\tilde{\mathbf{h}’} = \beta\mathbf{q} + (1-\beta) \mathbf{n}_j$ and $\beta \in (0, 0.5)$. Text Augmentation # Lexical Edits # Easy Data Augmentation (EDA; Wei & Zou 2019 ) defines a set of simple but powerful operations for text augmentation. Given a sentence, EDA randomly chooses and applies one of four simple operations: Synonym replacement (SR): Replace $n$ random non-stop words with their synonyms. Random insertion (RI): Place a random synonym of a randomly selected non-stop word in the sentence at a random position. Random swap (RS): Randomly swap two words and repeat $n$ times. Random deletion (RD): Randomly delete each word in the sentence with probability $p$. where $p=\alpha$ and $n=\alpha \times \text{sentence_length}$, with the intuition that longer sentences can absorb more noise while maintaining the original label. The hyperparameter $\alpha$ roughly indicates the percent of words in one sentence that may be changed by one augmentation. EDA is shown to improve the classification accuracy on several classification bench