Meta-RL with Outer and Inner Optimization Loops

Paper context

Paper title: Meta Reinforcement Learning | Lil'Log Abstract: The figure illustrates the meta-RL architecture, distinguishing between the inner loop of agent-environment interaction and the outer loop of meta-parameter optimization across a distribution of environments. Source body: Table of Contents On the Origin of Meta-RL Back in 2001 Proposal in 2016 Define Meta-RL Formulation Main Differences from RL Key Components Meta-Learning Algorithms for Meta-RL Optimizing Model Weights for Meta-learning Meta-learning Hyperparameters Meta-learning the Loss Function Meta-learning the Exploration Strategies Episodic Control Training Task Acquisition Task Generation by Domain Randomization Evolutionary Algorithm on Environment Generation Learning with Random Rewards References In my earlier post on meta-learning , the problem is mainly defined in the context of few-shot classification. Here I would like to explore more into cases when we try to “meta-learn” Reinforcement Learning (RL) tasks by developing an agent that can solve unseen tasks fast and efficiently. To recap, a good meta-learning model is expected to generalize to new tasks or new environments that have never been encountered during training. The adaptation process, essentially a mini learning session , happens at test with limited exposure to the new configurations. Even without any explicit fine-tuning (no gradient backpropagation on trainable variables), the meta-learning model autonomously adjusts internal hidden states to learn. Training RL algorithms can be notoriously difficult sometimes. If the meta-learning agent could become so smart that the distribution of solvable unseen tasks grows extremely broad, we are on track towards general purpose methods — essentially building a “brain” which would solve all kinds of RL problems without much human interference or manual feature engineering. Sounds amazing, right? 💖 On the Origin of Meta-RL # Back in 2001 # I encountered a paper written in 2001 by Hochreiter et al. when reading Wang et al., 2016 . Although the idea was proposed for supervised learning, there are so many resemblances to the current approach to meta-RL. The meta-learning system consists of the supervisory and the subordinate systems. The subordinate system is a recurrent neural network that takes as input both the observation at the current time step, $x\_t$ and the label at the last time step, $y\_{t-1}$. (Image source: Hochreiter et al., 2001 ) Hochreiter’s meta-learning model is a recurrent network with LSTM cell. LSTM is a good choice because it can internalize a history of inputs and tune its own weights effectively through BPTT . The training data contains $K$ sequences and each sequence is consist of $N$ samples generated by a target function $f_k(.), k=1, \dots, K$, $$ \{\text{input: }(\mathbf{x}^k_i, \mathbf{y}^k_{i-1}) \to \text{label: }\mathbf{y}^k_i\}_{i=1}^N \text{ where }\mathbf{y}^k_i = f_k(\mathbf{x}^k_i) $$ Noted that the last label $\mathbf{y}^k_{i-1}$ is also provided as an auxiliary input so that the function can learn the presented mapping. In the experiment of decoding two-dimensional quadratic functions, $a x_1^2 + b x_2^2 + c x_1 x_2 + d x_1 + e x_2 + f$, with coefficients $a$-$f$ are randomly sampled from [-1, 1], this meta-learning system was able to approximate the function after seeing only ~35 examples. Proposal in 2016 # In the modern days of DL, Wang et al. (2016) and Duan et al. (2017) simultaneously proposed the very similar idea of Meta-RL (it is called RL^2 in the second paper). A meta-RL model is trained over a distribution of MDPs, and at test time, it is able to learn to solve a new task quickly. The goal of meta-RL is ambitious, taking one step further towards general algorithms. Define Meta-RL # Meta Reinforcement Learning , in short, is to do meta-learning in the field of reinforcement learning . Usually the train and test tasks are different but drawn from the same family of problems; i.e., experiments in the papers included multi-armed bandit with different reward probabilities, mazes with different layouts, same robots but with different physical parameters in simulator, and many others. Formulation # Let’s say we have a distribution of tasks, each formularized as an MDP (Markov Decision Process), $M_i \in \mathcal{M}$. An MDP is determined by a 4-tuple, $M_i= \langle \mathcal{S}, \mathcal{A}, P_i, R_i \rangle$: Symbol Meaning $\mathcal{S}$ A set of states. $\mathcal{A}$ A set of actions. $P_i: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}_{+}$ Transition probability function. $R_i: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ Reward function. (RL^2 paper adds an extra parameter, horizon $T$, into the MDP tuple to emphasize that each MDP should have a finite horizon.) Note that common state $\mathcal{S}$ and action space $\mathcal{A}$ are used above, so that a (stochastic) policy: $\pi_\theta: \mathcal{S} \times \mathcal{A} \to \mathbb{R}_{+}$ would get inputs compatible across different tasks. The test tasks are sampled from the same distribution $\mathcal{M}$ or slightly modified version. Illustration of meta-RL, containing two optimization loops. The outer loop samples a new environment in every iteration and adjusts parameters that determine the agent's behavior. In the inner loop, the agent interacts with the environment and optimizes for the maximal reward. (Image source: Botvinick, et al. 2019 ) Main Differences from RL # The overall configure of meta-RL is very similar to an ordinary RL algorithm, except that the last reward $r_{t-1}$ and the last action $a_{t-1}$ are also incorporated into the policy observation in addition to the current state $s_t$. In RL: $\pi_\theta(s_t) \to$ a distribution over $\mathcal{A}$ In meta-RL: $\pi_\theta(a_{t-1}, r_{t-1}, s_t) \to$ a distribution over $\mathcal{A}$ The intention of this design is to feed a history into the model so that the policy can internalize the dynamics between states, rewards, and actions in the current MDP and adjust its strategy accordingly. This is well aligned with the setup in Hochreiter’s system . Both meta-RL and RL^2 implemented an LSTM policy and the LSTM’s hidden states serve as a memory for tracking c