Projector vs Phoneme Speech-Language Interfaces

Paper context

Paper title: Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR Abstract: Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector. Passages referencing this figure: rely been studied side by side under controlled backbones. This work presents a systematic study of speech–language interface design for LLM-ASR. We compare projector-based and phoneme-based interfaces in both high-resource and low-resource settings. We further propose a BPE-phoneme interface that merges frequent local phoneme patterns while retaining explicit word boundaries for the LLM decoder. Figure 1: Overview of the two types of speech–language interfaces studied in this work. (a) Projector-based: a trainable projector transforms downsampled speech-encoder ouput vectors into the LLM embedding space, followed by LoRA adaptation of the LLM. (b) Phoneme-based: we first train a CTC speech-to-phoneme (S2P) model, then feed sampled phoneme sequences into the LLM and adapt it with LoRA for termediate discrete interface between an S2P model and an LLM-based P2G model. While phoneme supervision has shown promise in multilingual ASR pretraining (e.g., Whistle [ yusuyin2025whistle ] ), controlled comparisons of phoneme-based and projector-based interfaces within the same LLM-ASR backbone remain limited; this work aims to fill that gap. 3 Method 3.1 Projector-based Interface As shown in Fig. 1 (a), the projector-interface approach consists of a speech encoder, a trainable projector, and an auto-regressive LLM decoder. For each utterance, we denote the speech signal as 𝐱 \mathbf{x} and the target transcript as 𝐲 \mathbf{y} . The speech encoder extracts frame-level acoustic representations: 𝐇 𝐱 = Enc ​ ( 𝐱 ) , \mathbf{H}_{\mathbf{x}}=\mathrm{Enc}(\mathbf{x}), (1) where 𝐇 𝐱 ∈ ℝ T × 𝐄 x ∈ ℝ N × d \mathbf{E}_{x}\in\mathbb{R}^{N\times d} serves as the speech-conditioned ``prefix'' embeddings for the LLM to generate the transcript. In our experiments we evaluate both vanilla project