Sequential CNN and Recurrent Cell Representations

Paper context

Paper title: Neural Architecture Search | Lil'Log Abstract: The figure illustrates how CNN layer hyperparameters and recurrent cell operations can be encoded as sequential strings for neural architecture search. Source body: Table of Contents Search Space Sequential Layer-wise Operations Cell-based Representation Hierarchical Structure Memory-bank Representation Search Algorithms Random Search Reinforcement Learning Evolutionary Algorithms Progressive Decision Process Gradient descent Evaluation Strategy Training from Scratch Proxy Task Performance Parameter Sharing Prediction-Based One-Shot Approach: Search + Evaluation What’s the Future? Appendix: Summary of NAS Papers Citation Reference Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures. Automatically learning and evolving network topologies is not a new idea ( Stanley & Miikkulainen, 2002 ). In recent years, the pioneering work by Zoph & Le 2017 and Baker et al. 2017 has attracted a lot of attention into the field of Neural Architecture Search (NAS), leading to many interesting ideas for better, faster and more cost-efficient NAS methods. As I started looking into NAS, I found this nice survey very helpful by Elsken, et al 2019 . They characterize NAS as a system with three major components, which is clean & concise, and also commonly adopted in other NAS papers. Search space : The NAS search space defines a set of operations (e.g. convolution, fully-connected, pooling) and how operations can be connected to form valid network architectures. The design of search space usually involves human expertise, as well as unavoidably human biases. Search algorithm : A NAS search algorithm samples a population of network architecture candidates. It receives the child model performance metrics as rewards (e.g. high accuracy, low latency) and optimizes to generate high-performance architecture candidates. Evaluation strategy : We need to measure, estimate, or predict the performance of a large number of proposed child models in order to obtain feedback for the search algorithm to learn. The process of candidate evaluation could be very expensive and many new methods have been proposed to save time or computation resources. Three main components of Neural Architecture Search (NAS) models. (Image source: Elsken, et al. 2019 with customized annotation in red) Search Space # The NAS search space defines a set of basic network operations and how operations can be connected to construct valid network architectures. Sequential Layer-wise Operations # The most naive way to design the search space for neural network architectures is to depict network topologies, either CNN or RNN, with a list of sequential layer-wise operations , as seen in the early work of Zoph & Le 2017 & Baker et al. 2017 . The serialization of network representation requires a decent amount of expert knowledge, since each operation is associated with different layer-specific parameters and such associations need to be hardcoded. For example, after predicting a conv op, the model should output kernel size, stride size, etc; or after predicting an FC op, we need to see the number of units as the next prediction. (Top) A sequential representation of CNN. (Bottom) A sequential representation of the tree structure of a recurrent cell. (Image source: Zoph & Le 2017 ) To make sure the generated architecture is valid, additional rules might be needed ( Zoph & Le 2017 ): If a layer is not connected to any input layer then it is used as the input layer; At the final layer, take all layer outputs that have not been connected and concatenate them; If one layer has many input layers, then all input layers are concatenated in the depth dimension; If input layers to be concatenated have different sizes, we pad the small layers with zeros so that the concatenated layers have the same sizes. The skip connection can be predicted as well, using an attention -style mechanism. At layer $i$ , an anchor point is added with $i−1$ content-based sigmoids to indicate which of the previous layers to be connected. Each sigmoid takes as input the hidden states of the current node $h_i$ and $i-1$ previous nodes $h_j, j=1, \dots, i-1$ . $$ P(\text{Layer j is an input to layer i}) = \text{sigmoid}(v^\top \tanh(\mathbf{W}_\text{prev} h_j + \mathbf{W}_\text{curr} h_i)) $$ The sequential search space has a lot of representation power, but it is very large and consumes a ton of computation resources to exhaustively cover the search space. In the experiments by Zoph & Le 2017 , they were running 800 GPUs in parallel for 28 days and Baker et al. 2017 restricted the search space to contain at most 2 FC layers. Cell-based Representation # Inspired by the design of using repeated modules in successful vision model architectures (e.g. Inception, ResNet), the NASNet search space ( Zoph et al. 2018 ) defines the architecture of a conv net as the same cell getting repeated multiple times and each cell contains several operations predicted by the NAS algorithm. A well-designed cell module enables transferability between datasets. It is also easy to scale down or up the model size by adjusting the number of cell repeats. Precisely, the NASNet search space learns two types of cells for network construction: Normal Cell : The input and output feature maps have the same dimension. Reduction Cell : The output feature map has its width and height reduced by half. The NASNet search space constrains the architecture as a repeated stack of cells. The cell architecture is optimized via NAS algorithms. (Image source: Zoph et al. 2018 ) The predictions for each cell are grouped into $B$ blocks ($B=5$ in the NASNet paper), where each block has 5 prediction steps made by 5 distinct softmax classifiers corresponding to discrete choices of the elements of a block. Note that the NASNet search space does not have residual connections be