{"ID":5439532,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-02T21:49:13.561239862Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.30944","arxiv_id":"2606.30944","title":"Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation","abstract":"Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.","short_abstract":"Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-...","url_abs":"https://arxiv.org/abs/2606.30944","url_pdf":"https://arxiv.org/pdf/2606.30944v1","authors":"[\"Yuxuan Hu\",\"Heng Lu\",\"Ruchao Fan\",\"Yao Qian\",\"Xiaofei Wang\",\"Jian Xue\",\"Heming Wang\",\"Shuohang Wang\",\"Young Jin Kim\",\"Yelong Shen\",\"Jinyu Li\"]","published":"2026-06-29T21:55:18Z","proceeding":"eess.AS","tasks":"[\"eess.AS\",\"cs.SD\"]","methods":"[\"Large Language Model\"]","has_code":false}