Joint 1-D and 2-D Audio Modeling Framework Schematic

论文上下文

Paper title: ULTRAS -- Unified Learning of Transformer Representations for Audio and Speech Signals Abstract: Self-supervised learning (SSL) has driven impressive advances in speech processing by adopting time-domain prediction objectives, while audio representation learning frameworks operate on time-frequency spectrograms. Models optimized for one paradigm struggle to transfer to the other, highlighting the need for a joint framework. We propose Unified Learning of Transformer Representations for Audio and Speech (ULTRAS), where the masking and predictive modeling is performed over long patches of the data. The model, based on the transformer architecture, encodes spectral-patches of log-mel spectrogram features. The predictive modeling of masked segments is performed on spectral and temporal targets using a combined loss-function, forcing the representations to encode time and frequency traits. Experiments are performed on a variety of speech and audio tasks, where we illustrate that the ULTRAS framework achieves improved performance over other established baselines. Passages referencing this figure: rame-level representation learning was proposed by Xian et al. [ 13 ] . In another work, Nilzumi et al. [ 15 ] proposed the learning of separate models for masked and unmasked regions of the audio. However, none of these frameworks explicitly model the joint temporal and spectral information present in the audio signal, leading to a degradation in the diverse evaluation of speech and audio tasks. Figure 1: Block schematic of the proposed framework of joint 1-D and 2-D modeling of audio data. The gradient colored blocks are learnable, while the rest do not have any learnable parameters. In this paper, we propose U nified L earning of T ransformer R epresentations for A udio and S peech (ULTRAS), an approach to jointly model the time-frequency attributes of the input acoustic signal. Unlike e key contributions of this work are:- • Masked modeling of long audio segments of syllable length for effective encoding of acoustic information. • Joint prediction of spectral and temporal targets for embedding time-frequency characteristics. • Comprehensive evaluation on a diverse set of speech and audio downstream tasks to illustrate the effectiveness of the work. II Proposed ULTRAS Framework Figure 1 illustrates the main workflow of our proposed iterative audio pre-training framework. Our approach leverages a Vision Transformer-style SSL model that takes a 2-D audio spectrogram as input. The model is optimized through an unified loss function. The goal is to learn robust and generalizable audio representations applicable to both speech and audio tasks. II-A Input pre-processing The in n of the spectral loss ℒ s \mathcal{L}_{\text{s}} and time-frame loss ℒ t \mathcal{L}_{\text{t}} : ℒ total = λ ​ ℒ t + ( 1 − λ ) ​ ℒ s \mathcal{L}_{\text{total}}=\lambda\mathcal{L}_{\texttt{t}}+(1-\l