SLLM Training with Bias Word Position Tag Prediction

Paper context

Paper title: Contextual Biasing for ASR in Speech LLM with Common Word Cues and Bias Word Position Prediction Abstract: Speech-aware LLMs (SLLMs) have recently achieved state-of-the-art ASR performance; however, they still fail to accurately transcribe bias words that appear rarely or never in the training data. Contextual biasing mechanisms are commonly implemented by introducing a predefined bias word list into the model via a text prompt or additional module. For further improvement, predefined bias words can be paired with their phoneme representations as pronunciation cues. Typically, phoneme sequences are generated through a G2P system that covers the target languages and domains of the bias words. Therefore, when a compatible G2P system is unavailable, phoneme-assisted contextual biasing becomes difficult to perform. Moreover, manually adding accurate phoneme sequences requires advanced phonetic knowledge. In this paper, we explore contextual biasing in SLLM based on acoustic cues associated with a set of common words whose pronunciations are partially similar to those of the target bias words. We assume ASR applications in which end users do not require special knowledge of phonetics or utilize G2P tools for inference. For enhanced robustness, we also introduce bias word positional prediction implemented in a multi-output learning fashion. Our method reduces bias word recognition errors by 16.3% compared to baseline systems, including on out-of-domain data. Passages referencing this figure: riments. 2.3 Proposed bias word position prediction To enhance the SLLM’s robustness on contextual ASR tasks without degradation on non-contextual ASR tasks, we train our model through a multi-task and multi-output training framework. Multi-task training is performed by training the SLLM on prompts of multiple target tasks together. On the other hand, the proposed multi-output training mechanism (Fig. 1 ) trains the SLLM to predict the positions of bias words within an utterance, in addition to speech transcription. The bias word positions are represented as a sequence of character-level tags W = [ w 1 , w 2 , … , w V ] W=[w_{1},w_{2},...,w_{V}] consisting of V V tokens, where each is associated with a character in the speech transcription. Our tag set consists of the “ bias ”, “ non-bias ) is as follows: L S ​ L ​ L ​ M = L A ​ S ​ R ​ ( T , T ^ ) + α ​ L C ​ T ​ C ​ ( W , W ^ ) , L_{SLLM}=L_{ASR}(T,\hat{T})+\alpha L_{CTC}(W,\hat{W}),\vskip-8.5359pt (3) where L A ​ S ​ R L_{ASR} is the loss for the speech recognition task, T ^ \hat{T} is the speech transcription output, W ^ \hat{W} is the bias word positional tag output, and α \alpha is the loss coefficient for bias word tagger. Fig. 1 : The proposed SLLM training with a bias word position tag prediction mechanism. The bias list applies the proposed cues using syllable-based partial phonetic matching. (“ b ”: character-level bias word tag, “ n ”: character-level non-bias word tag) 3 Experiment setting 3.1 Model We used the Granite-Speech [ 23 ] 1 1 1 https://huggingface.co/ibm-granite/granite-speech-3.3-8b architecture as