Sema Token-Based Speech Coding Architecture Comparison
Figure 3 . Architectural comparison. (a) Traditional pipelines send perceptually coded media over the network ( ∼ \sim 12–700 KB per turn) on both paths. (b) Sema tokenizes on the client (§ 3.1 ), reconstructs on the server (§ 3.2 ), and decodes speech tokens via a client-side vocoder on the downlink (§ 3.3 ). Both directions share a lightweight token framing protocol.
论文上下文
Paper title: Sema: Semantic Transport for Real-Time Multimodal Agents Abstract: Real-time multimodal agents transport raw audio and screenshots using networking stacks designed for human receivers, which optimize for perceptual fidelity and smooth playout. Yet agent models act as event-driven processors with no inherent sense of physical time, consuming task-relevant semantics rather than reconstructing signals in real time. This fundamental difference shifts the transport goal from the technical problem of signal fidelity (Shannon-Weaver Level A) to the semantic problem of meaning preservation (Level B). This mismatch imposes significant overhead. In visual pipelines, screenshot upload accounts for over 60% of end-to-end action latency on constrained uplinks, and in voice pipelines, conventional transport carries massive redundancy, sending 43-64x more data than needed to maintain task accuracy. We present Sema, a semantic transport system that combines discrete audio tokenizers with a hybrid screen representation (lossless accessibility-tree or OCR text, plus compact visual tokens) and bursty token delivery that eliminates jitter buffers. In simulations under emulated WAN conditions, Sema reduces uplink bandwidth by 64x for audio and 130-210x for screenshots while preserving task accuracy within 0.7 percentage points of the raw baseline. Passages referencing this figure: h Opus at 32 kbps. In contrast, a semantic layer using SpeechTokenizer (Zhang et al. , 2024 ) produces just 50 discrete tokens/s ( ∼ \sim 62.5 B/s), achieving a ∼ \sim 64 × \times reduction (Figure 2 ). This massive compression preserves downstream ASR accuracy because the first RVQ layer retains linguistic content (Défossez et al. , 2023 ) , effectively isolating semantics from acoustic details. Figure 1 . Per-turn uplink bytes (log scale). Grouped bar chart on log scale comparing per-turn uplink bytes for Raw, Compress, and Sema tokens. Figure 2 . Screenshot latency vs. uplink bandwidth. Line plot of screenshot latency versus uplink bandwidth. 2.2. Why Semantic Transport Works The magnitude of these reductions stems not from more aggressive lossy compression but from optimizing a fundame
