TASU2 Framework Overview

Paper context

Paper title: TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs Abstract: Speech LLM post-training increasingly relies on efficient cross-modal alignment and robust low-resource adaptation, yet collecting large-scale audio-text pairs remains costly. Text-only alignment methods such as TASU reduce this burden by simulating CTC posteriors from transcripts, but they provide limited control over uncertainty and error rate, making curriculum design largely heuristic. We propose \textbf{TASU2}, a controllable CTC simulation framework that simulates CTC posterior distributions under a specified WER range, producing text-derived supervision that better matches the acoustic decoding interface. This enables principled post-training curricula that smoothly vary supervision difficulty without TTS. Across multiple source-to-target adaptation settings, TASU2 improves in-domain and out-of-domain recognition over TASU, and consistently outperforms strong baselines including text-only fine-tuning and TTS-based augmentation, while mitigating source-domain performance degradation. Passages referencing this figure: C simulator that generates posterior sequences under a specified WER range, enabling explicit control over supervision difficulty and error profiles for curriculum design. • We demonstrate consistent gains over TASU on both source-domain and generalization evaluations without audio training, and competitive improvements in low-resource transfer over text-only and TTS-based augmentation baselines. Figure 1: An Overview of TASU2. 2 Text-only Alignment: From TASU to TASU2 CTC introduces a blank symbol and marginalizes over alignments, collapsing frame-level posteriors into compact label sequences via blank removal and repetition merging [ jung2023blankcollapsecompressingctc , deng2021improvinghybridctcattentionendtoend ] . This compact representation has inspired several speech-LLM alignment nly alignment is whether a simulator can generate CTC-like posteriors that are both (i) close to real acoustic posteriors and (ii) controllable to support principled curricula and low-resource adaptation . We address this with TASU2 , which learns a controllable text-to-CTC posterior simulator that outputs pseudo CTC posterior sequences conditioned on a transcript and a discrete WER control code. Fig. 1 overviews the pipeline. 3.1 Task and Notation Let the transcript be a token sequence 𝐲 = ( y 1 , … , y U ) \mathbf{y}=(y_{1},\ldots,y_{U}) from a tokenizer with vocabulary size V V (including the CTC blank). TASU2 learns a simulator that maps ( 𝐲 , c ) (\mathbf{y},c) to a posterior-frame sequence 𝐏 ^ = ( 𝐩 ^ 1 , … , 𝐩 ^ T ) \hat{\mathbf{P}}=(\hat{\mathbf{p}}_{1},\ldots,\hat{\mathbf{p}}_{T}) um_{t=1}^{T_{\text{train}}}m_{t}\sum_{v=1}^{V}\mathbf{p}_{t,v}\log\hat{\mathbf{p}}_{t,v}\right]. (3) This distribution-matching objective encourages CTC-like structure like blank dominance and token