Pangea Unified Semantic Space for Actions — Poster

Paper context

Paper title: From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding Abstract: This poster presents Pangea, a unified semantic space for human action understanding constructed from 18 datasets. It details the P2S method mapping physical to semantic space and demonstrates performance improvements on benchmarks like HICO and Kinetics-400. Paper body (method & results): From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding Yong-Lu Li*, Xiaoqian Wu*, Xinpeng Liu, Zehao Wang, Yiming Dou, Yikun Ji, Junyi Zhang, Yixing Li, Jingru Tan, Xudong Lu, Cewu Lu† Shanghai Jiao Tong University {yonglu li, enlighten, davidwang200099, douyiming, junyizhang, lyxing0, luxudong2001, lucewu}@sjtu.edu.cn, {xinpengliu0907, jiyikun2002, tanjingru120}@gmail.com Abstract Action understanding has attracted long-term attention. It can be formed as the mapping from the physical space to the semantic space. Typically, researchers built datasets according to idiosyncratic choices to define classes and push the envelope of benchmarks respectively. Datasets are incompatible with each other like “Isolated Islands” due to semantic gaps and various class granularities, e.g., do housework in dataset A and wash plate in dataset B. We argue that we need a more principled semantic space to concentrate the community efforts and use all datasets together to pursue generalizable action learning. To this end, we design a structured action semantic space given verb taxonomy hierarchy and covering massive actions. By aligning the classes of previous datasets to our semantic space, we gather (image/video/skeleton/MoCap) datasets into a unified database in a unified label system, i.e., bridg- ing “isolated islands” into a “Pangea”. Accordingly, we propose a novel model mapping from the physical space to semantic space to fully use Pangea. In extensive experi- ments, our new system shows significant superiority, espe- cially in transfer learning. Our code and data will be made public at https://mvig-rhos.com/pangea. 1. Introduction Visual action understanding is an important direction in computer vision and matters to various domains [21, 106]. Generally speaking, it can be formulated as the mapping from the physical space to the semantic space. Here, physi- cal space indicates the visual patterns (information carrier) and semantic space represents the action semantics (class). In terms of the physical space, many works were pro- posed to extract representations from different modalities *The first two authors contribute equally. †Corresponding author. HICO DET sit on bench Kinetics playing violin HAG put / pour sth in kettle Babel kick HAA CPR Stanford40 phoning HAG wash bowl Figure 1. “Isolated islands”. The semantic gap brings a great chal- lenge to general action understanding. to capture action cues, such as image [58], video [8], skele- ton [38], MoCap [34], RGBD [100], etc. However, few efforts have been made to semantic space design. Previ- ous benchmarks [9, 15, 44] are typically designed accord- ing to designers’ choice and incompatible with each other due to semantic gaps. They have three main weaknesses: (1) Ambiguity. Similar actions may have different class names, e.g., clean, wipe, scrub. Though this may strengthen the diversity in visual-language learning [94], it hinders machines from learning the subtle similarities and differences of actions. Besides, the same class may rep- resent different actions, e.g., address means either ad- dressing oneself to something or addressing a conference. This phenomenon brings both generalization possibility and challenge. (2) Overlooking granularity/hierarchy. The datasets are constructed independently, thus typically over- looking granularity, e.g., do housework in dataset A and clean floor in dataset B, sometimes even in one dataset. (3) Integration/transfer difficulty. Large models need more data. However, due to the “isolated islands”, it is hard to integrate datasets and conclude the “few-shotness” and “zero-shotness” of classes. We do not know which arXiv:2304.00553v4 [cs.CV] 3 Apr 2024 hold hug pet scratch …… hug pet scratch scratch hug hold hold Action Semantics VerbNet Semantics Semantic Knowledge Touch-20-1 Verb Members: grasp, massage… Example Sentences: touched on… massage: <Manipulation> massage%2:35:00 Definitions: manually manipulate… Examples: She rubbed down… Lexname: verb.contact massage%2:29:00… grasp: <Manipulation>…… Geometric Knowledge mapping Figure 2. Verb tree. The conventional action semantics (e.g., hold, hug) can be mapped into node semantics (e.g., touch-20-1, support-15.3). The proposed semantic space has abundant semantic and geometric knowledge. classes should be enriched or used for transfer learning. In Fig. 1, we visualize the class word embeddings [28] of 18 datasets via t-SNE. Huge semantic gaps exist. Even for the very large Kinetics-700 [8], there are still many classes beyond its coverage. Here, we first clearly reveal the overlooked “Isolated Islands (I2)” problem. It brings semantic gaps and impedes cross-dataset learning. Though CLIP [94]-like works alleviate this problem to some ex- tent with the open-vocabulary property, their latent space may be difficult to capture the subtle polysemy, taxonomy, and hierarchy of action semantics. In experiments (Sec. 6), CLIP trained with simply-mixed datasets performs not well. Thus, we rethink the action semantic space design and take a step towards a principled semantic space. We pro- pose a new system to pave a promising way to address the I2 problem. Our core idea is to use a structured ac- tion semantic space to replace the existing hand-crafted ones. We build this semantic space according to the lin- guistic structure knowledge of VerbNet [99]. VerbNet is a network linking the syntactic and semantic patterns of verbs. It is a domain-independent tree-structure lexicon and has a clear hierarchy covering most verbs. We vi- sualize the verb tree in Fig. 2. To maximize the poten- tial of our semantic space, we gather many datasets (im- age/video/skeleton/MoCap) to build a database and align their classes to our semantic space easily, i.e., linking the “isolated islands” into a “Pangea”. Then, we can use the continuous hyperbolic space together with the semantic- geometric prompt to embed the structured knowledge. Our space has four-fold superiority: (1) Unambigu- ous verb nodes correlating all related verbs, e.g., pat, nudge, massage with similar meaning are shared by the node touch-20-1. (2) Rich knowledge. Besides the thematic role, syntactic, semantic description, and se- lectional preferences of verbs, VerbNet has mappings to other knowledge bases (WordNet [80], PropBank [46], FrameNet [3]). We can conveniently adopt Large Language Models [5] to extract meaningful language representations to advance learning. (3) Hierarchy to represent actions from abstract to specific granularity, e.g., sports, ball sports, basketball, dunk. (4) Extensive cover- age. It contains about 5,800 verbs. In Fig. 1, our space not only covers all datasets but also spans the semantics a lot. To fully use Pangea, we propose a compact mapping system to conduct action understanding, which effectively maps multi-modal physical patterns to the structured se- mantic space. In experiments, our method armed with Pangea demonstrates representative and transfer ability. On multi-modal benchmarks, it brings decent improvements. Our contributions are: 1) We propose a structured se- mantic space to bridge the “isolated islands”. 2) We build the Pangea database gathering 28 multi-modal datasets. 3) A physical-to-semantic mapping model is proposed given Pangea and shows significant transfer ability. 2. Related Work Action Understanding has achieved progress recently. There are mainly image [9, 35, 128], video [22, 32, 44, 107], skeleton [63], and 3D body [92] datasets. The common tasks are action recognition and temporal/spatial localiza- tion/detection. Early benchmarks focus on classifying an image or a short video into classes [9, 107, 128]. Re- cently, benchmarks that require both accurate recognition and active subject detection are emerging [10, 32, 35, 56, 65, 66, 122]. Moreover, few/zero-shot action learning [12] also attracts attention. Many methods have been pro- posed to push this direction forward. For image tasks, 2D CNN is the dominant architecture, while knowledge like part state [57, 73], 2D/3D human [54, 55], and language prior [4, 37, 58, 90] is used too. For video tasks, 2D- CNN [20, 59, 130], two-stream network [24, 104], and 3D- CNN [8, 25? ] are the major architectures adopted. For skeleton tasks, both GCN [53, 68, 125] and 2D-CNN [13, 124] are widely used. Recently, with the success of Trans- former [117, 123], besides directly importing it into action detection [7, 112], visual-language contrastive learning [94] has changed this direction a lot. In terms of action semantic space, most datasets [9, 32, 35, 48, 50] overlook action hierarchy. While some works consider hierarchy [22, 69, 101]. For exam- ple, ActivityNet [22] defines 200+ action classes be- longing to 7 high-level classes (e.g., personal care, household) based on activity scenarios; FineGym [101] organizes hierarchical actions from gymnasium videos; VerSe [30] augments COCO [60] and TUHOI [51] with verb sense labels to provide finer-grained action semantics on 3.5 K images. However, they are scale/class/domain- limited and built with manually-picked classes. Instead, we choose to cover the hierarchy based on well-defined linguistic works such as VerbNet [99], WordNet [80], FrameNet [3], etc. 3. Preliminary In this section, we first introduce the preliminaries of the physical and semantic space. Multi-Modal Physical Space. Here, we adopt two modalities for physical space P: 2D and 3D. For 2D, we adopt CNN or Transformer (e.g., ResNet [36], CLIP [94]) to extract representation from image/video. For 3D, we use the widely-used model SMPL [70] to embed 3D humans. Structured Semantic Space. Intuitively, the ambiguity of objects is relatively smaller, thus objects/nouns are eas- ier to label. Things are different for actions/verbs which are more ambiguous. Previous works typically design se- mantic space manually and optionally. Instead, we build the structured semantic space S via the hierarchical verb tree from VerbNet [99] (Fig. 2). Here, we define the nodes as the classes of our semantic space. Compared with con- ventional design [22], our space has elegant characteristics: (1) Due to the lack of a unified naming standard, classes of previous datasets have ambiguity. For example, differ- ent datasets may have feast, eating, and dining re- spectively, where a common semantic is shared. Instead, in our S, actions with shared meanings are connected with their common nodes. (2) Each node is equipped with abun- dant knowledge. In Fig. 2, touch-20-1 node is ex- plained by: a) Verb members, e.g., grasp; b) Example sentences as instantiations of the node semantics; c) Each verb member is explained via connections with other lexical resources (e.g., WordNet [80], FrameNet [3]). In Fig. 2, the verb massage is explained by its frame in FrameNet [3] (manipulation) and the corresponding items in Word- Net [80] (massage%2:35:00, massage%2:29:00). (3) Hierarchy reveals semantic connections between