GROOT Self-Supervised Pre-Training in Minecraft — Poster

Paper context

Paper title: GROOT: Learning to Follow Instructions by Watching Gameplay Videos Abstract: A conference poster detailing the GROOT framework for self-supervised pre-training in Minecraft. It covers architecture design, goal space visualization via t-SNE, the SkillForge benchmark results, and chain of instruction capabilities. Paper body (method & results): GROOT: Learning to Follow Instructions by Watching Gameplay Videos Shaofei Cai1, Bowei Zhang1, Zihao Wang1, Xiaojian Ma3, Anji Liu2 and Yitao Liang1 1PKU, 2UCLA, 3BIGAI, All authors are affiliated with Team CraftJarvis, We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis. The project page is available at https: //craftjarvis-groot.github.io. Figure 1 | Through the cultivation of extensive gameplay videos, GROOT has grown a rich set of skill fruits (number denotes success rate; skills shown above do not mean to be exhaustive; kudos to our artist Haowei). 1. Introduction Developing human-level embodied agents that can solve endless tasks in open-world environments, such as Minecraft (Fan et al., 2022; Johnson et al., 2016), has always been a long-term goal pursued in AI. Recent works have explored using Large Language Models (LLMs) to generate high-level plans, which guide the agent to accomplish challenging long-horizon tasks (Wang et al., 2023a,b; Zhu et al., 2023). However, a major gap between these LLM-based agents and generalist agents that can complete endless amounts of tasks is the capability of their low-level controllers, which map the plans to motor commands. Recently developed controllers are only capable of completing a predefined and narrow set of programmatic tasks (Baker et al., 2022; Cai et al., 2023; Lin et al., 2021), which hinders LLM-based planning agents from unleashing their full potential. We attribute the limitation of these low-level controllers to how the goal is specified. Specifically, existing controllers use task indicator Corresponding author(s): Yitao Liang Shaofei Cai<caishaofei@stu.pku.edu.cn>, Bowei Zhang<zhangbowei@stu.pku.edu.cn>,Zihao Wang<zhwang@stu.pku.edu.cn>, Xiaojian Ma<xiaojian.ma@ucla.edu>, Anji Liu<liuanji@cs.ucla.edu>, Yitao Liang<yitaol@pku.edu.cn> arXiv:2310.08235v2 [cs.AI] 29 Nov 2023 GROOT: Learning to Follow Instructions by Watching Gameplay Videos (Yu et al., 2019), future outcome (Chen et al., 2021; Lifshitz et al., 2023), and language (Brohan et al., 2022) to represent the goal. While it is easy to learn a controller with some of these goal specifications, they may not be expressive enough for diverse tasks. Taking future outcome goals as an example, an image of a desired house clearly lacks procedural information on how the house was built. One exception is language, but learning a controller that can receive language goals is prohibitively expensive as it requires a huge number of trajectory-text pairs with text that precisely depicts the full details of the gameplay, therefore preventing them from scaling up to more open-ended tasks. Having observed the limitations of goal specification in the prior works, this paper seeks to find a balance between the capacity of goal specification and the cost of controller learning. Concretely, we propose to specify the goal as a reference gameplay video clip. While such video instruction is indeed expressive, there are two challenges: 1) How can the controller understand the actual goal being specified as the video itself can be ambiguous, i.e. a goal space or video instruction encoder has to be learned; 2) How to ultimately map such goal to actual motor commands? To this end, we introduce a learning framework that simultaneously produces a goal space and a video instruction following controller from gameplay videos. The fundamental idea is casting the problem as future state prediction based on past observations: • The predicting model needs to identify which goal is being pursued from the past observations, which requires a good goal space (induced by a video instruction encoder); • Since the transition dynamics model is fixed, a policy that maps both the state and the recognized goal to action is also needed by the predicting model when rolling the future state predictions. Effectively, this results in the goal space and control policy we need. We introduce a variational learning objective for this problem, which leads to a combination of a cloning loss and a KL regularization loss. Based on this framework, we implement GROOT, an agent with an encoder-decoder architecture to solve open-ended Minecraft tasks by following video instructions. The video encoder is a non-causal transformer that extracts the semantic information expressed in the video and maps it to the latent goal space. The controller policy is a decoder module implemented by a causal transformer, which decodes the goal information in the latent space and translates it into a sequence of actions in the given environment states in an autoregressive manner. To comprehensively evaluate an agent’s mastery of skills, we designed a benchmark called Minecraft SkillForge. The benchmark covers six common Minecraft task groups: collect, build, survive, explore, tool, and craft, testing the agent’s abilities in resource collection, structure building, environmental understanding, and tool usage, in a total of 30 tasks. We calculate Elo ratings among GROOT, several counterparts, and human players based on human evaluations. Our experiments showed that GROOT is closing the human-machine gap and outperforms the best baseline by 150 points (or 70% winning rate) in an Elo tournament system. Our qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis. To sum up, our main contributions are as follows: • Start by maximizing the log-likelihood of future states given past ones, we have discovered the learning objectives that lead to a good goal space and ultimately the instruction-following controller from gameplay videos. It provides theoretical guidance for the agent architecture design and model optimization. • Based on our proposed learning framework, we implemented a simple yet efficient encoder- decoder agent based on causal transformers. The encoder is responsible for understanding the goal information in the video instruction while the decoder as the policy emits motor commands. • On our newly introduced benchmark, Minecraft SkillForge, GROOT is closing the human-machine 2 GROOT: Learning to Follow Instructions by Watching Gameplay Videos gap and surpassing the state-of-the-art baselines by a large margin in the overall Elo rating com- parison. GROOT also exhibits several interesting emergent properties, including goal composition and complex gameplay behavior synthesis. 2. Preliminaries and Problem Formulation Reinforcement Learning (RL) concerns the problem in which an agent interacts with an environment at discrete time steps, aiming to maximize its expected cumulative reward (Espeholt et al., 2018; Mnih et al., 2015; Schulman et al., 2017). Specifically, the environment is defined as a Markov Decision Process (MDP) ⟨S, A, R, P, 𝑑0⟩, where S is the state space, A is the action space, R : S × A →ℝis the reward function, P : S × A →S is the transition dynamics, and 𝑑0 is the initial state distribution. Our goal is to learn a policy 𝜋(𝑎|𝑠) that maximizes the expected cumulative reward 𝔼[Í∞ 𝑡=0 𝛾𝑡𝑟𝑡], where 𝛾∈(0, 1] is a discount factor. In goal-conditioned RL (GCRL) tasks, we are additionally provided with a goal 𝑔∈G (Andrychowicz et al., 2017; Cai et al., 2023; Ding et al., 2019; Jing et al., 2020, 2021; Liu et al., 2022; Yang et al., 2019). And the task becomes learning a goal-conditioned policy 𝜋(𝑎|𝑠, 𝑔) that maximizes the expected return 𝔼[Í∞ 𝑡=0 𝛾𝑡𝑟𝑔 𝑡], where 𝑟𝑔 𝑡is the goal-specific reward achieved at time step 𝑡. Apart from being a new type of RL task, GCRL has been widely studied as a pre-training stage toward conquering more challenging environments/tasks (Aytar et al., 2018b; Baker et al., 2022; Zhang et al., 2022). Specifically, suppose we are provided with a good goal-condition policy, the goal can be viewed as a meta-action that drives the agent to accomplish various sub-tasks, which significantly simplifies tasks that require an extended horizon to accomplish. Further, when equipped with goal planners, we can achieve zero- or few-shot learning on compositional tasks that are beyond the reach of RL algorithms (Gong et al., 2023; Huang et al., 2022; Wang et al., 2023a,b; Zhu et al., 2023). At the heart of leveraging such benefits, a key requirement is to have a properly-defined goal space that (i) has a wide coverage of common tasks/behaviors, and (ii) succinctly describes the task without including unnecessary information about the state. Many prior works establish goal spaces using guidance from other modalities such as language (Cai et al., 2023; Hong et al., 2020; Stone et al., 2023) or code (Huang et al., 2023; Wang et al., 2023a). While effective, the requirement on large-scale trajectory data paired with this auxiliary information could be hard to fulfill in practice. Instead, this paper studies the problem of simultaneously learning a rich and coherent goal space and the corresponding goal-conditioned policy, given a pre-trained inverse dynamic model and raw gameplay videos, i.e. sequences of states {𝑠(𝑖) 0:𝑇}𝑖collected using unknown policies. 3. Goal Space Discovery via Future State Prediction This section explains our learning framework: discovering a “good” goal space as well as a video instruction following the controller through the task of predicting future states given previous ones. We start with an illustrative example in Minecraft (Johnson et al., 2016). Imagine that an agent is standing inside a grassland holding an axe that can be used to chop the tree in front of them. Suppose in the gameplay video, players either go straight to chop the tree or bypass it to explore the territory. In order to predict future frames, it is sufficient to know (i) which goal (chop tree or bypass tree) is being pursued by the agent, and (ii) what will happen if the agent chooses a particular option (i.e. transition dynamics). Apart from the latter information that is irrelevant to the past observations, we only need to capture the goal informat