How DeceptionNet Works: Domain Randomization Overview

Paper context

Paper title: Domain Randomization for Sim2Real Transfer | Lil'Log Abstract: The figure illustrates the two-step training process of DeceptionNet, showing how a deception network generates transformed inputs to fool a fixed recognition network, and then how these inputs are used to train the recognition network. Source body: Table of Contents What is Domain Randomization? Uniform Domain Randomization Why does Domain Randomization Work? DR as Optimization DR as Meta-Learning Guided Domain Randomization Optimization for Task Performance Match Real Data Distribution Guided by Data in Simulator References In Robotics, one of the hardest problems is how to make your model transfer to the real world. Due to the sample inefficiency of deep RL algorithms and the cost of data collection on real robots, we often need to train models in a simulator which theoretically provides an infinite amount of data. However, the reality gap between the simulator and the physical world often leads to failure when working with physical robots. The gap is triggered by an inconsistency between physical parameters (i.e. friction, kp, damping, mass, density) and, more fatally, the incorrect physical modeling (i.e. collision between soft surfaces). To close the sim2real gap, we need to improve the simulator and make it closer to reality. A couple of approaches: System identification System identification is to build a mathematical model for a physical system; in the context of RL, the mathematical model is the simulator. To make the simulator more realistic, careful calibration is necessary. Unfortunately, calibration is expensive. Furthermore, many physical parameters of the same machine might vary significantly due to temperature, humidity, positioning or its wear-and-tear in time. Domain adaptation Domain adaptation (DA) refers to a set of transfer learning techniques developed to update the data distribution in sim to match the real one through a mapping or regularization enforced by the task model. Many DA models, especially for image classification or end-to-end image-based RL task, are built on adversarial loss or GAN . Domain randomization With domain randomization (DR) , we are able to create a variety of simulated environments with randomized properties and train a model that works across all of them. Likely this model can adapt to the real-world environment, as the real system is expected to be one sample in that rich distribution of training variations. Both DA and DR are unsupervised. Compared to DA which requires a decent amount of real data samples to capture the distribution, DR may need only a little or no real data. DR is the focus of this post. Conceptual illustrations of three approaches for sim2real transfer. What is Domain Randomization? # To make the definition more general, let us call the environment that we have full access to (i.e. simulator) source domain and the environment that we would like to transfer the model to target domain (i.e. physical world). Training happens in the source domain. We can control a set of $N$ randomization parameters in the source domain $e_\xi$ with a configuration $\xi$, sampled from a randomization space, $\xi \in \Xi \subset \mathbb{R}^N$. During policy training, episodes are collected from source domain with randomization applied. Thus the policy is exposed to a variety of environments and learns to generalize. The policy parameter $\theta$ is trained to maximize the expected reward $R(.)$ average across a distribution of configurations: $$ \theta^* = \arg\max_\theta \mathbb{E}_{\xi \sim \Xi} [\mathbb{E}_{\pi_\theta, \tau \sim e_\xi} [R(\tau)]] $$ where $\tau_\xi$ is a trajectory collected in source domain randomized with $\xi$. In a way, “discrepancies between the source and target domains are modeled as variability in the source domain.” (quote from Peng et al. 2018 ). Uniform Domain Randomization # In the original form of DR ( Tobin et al, 2017 ; Sadeghi et al. 2016 ), each randomization parameter $\xi_i$ is bounded by an interval, $\xi_i \in [\xi_i^\text{low}, \xi_i^\text{high}], i=1,\dots,N$ and each parameter is uniformly sampled within the range. The randomization parameters can control appearances of the scene, including but not limited to the followings (see Fig. 2). A model trained on simulated and randomized images is able to transfer to real non-randomized images. Position, shape, and color of objects, Material texture, Lighting condition, Random noise added to images, Position, orientation, and field of view of the camera in the simulator. Images captured in the training environment are randomized. (Image source: Tobin et al, 2017 ) Physical dynamics in the simulator can also be randomized ( Peng et al. 2018 ). Studies have showed that a recurrent policy can adapt to different physical dynamics including the partially observable reality. A set of physical dynamics features include but are not limited to: Mass and dimensions of objects, Mass and dimensions of robot bodies, Damping, kp, friction of the joints, Gains for the PID controller (P term), Joint limit, Action delay, Observation noise. With visual and dynamics DR, at OpenAI Robotics, we were able to learn a policy that works on real dexterous robot hand ( OpenAI, 2018 ). Our manipulation task is to teach the robot hand to rotate an object continously to achieve 50 successive random target orientations. The sim2real gap in this task is very large, due to (a) a high number of simultaneous contacts between the robot and the object and (b) imperfect simulation of object collision and other motions. At first, the policy could barely survive for more than 5 seconds without dropping the object. But with the help of DR, the policy evolved to work surprisingly well in reality eventually. Why does Domain Randomization Work? # Now you may ask, why does domain randomization work so well? The idea sounds really simple. Here are two non-exclusive explanations I found most convincing. DR as Optimization # One idea ( Vuong, et al, 2019 ) is to view learning randomization parameters in DR as a bilevel optimization . Assuming we have access to the real environment $e_\text{real}$ and the randomization config is sampled from a distribution parameterized by $\phi$, $\xi \sim P_\phi(\xi)$, we would like to learn a dis