{"ID":2864903,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.02327","arxiv_id":"2510.02327","title":"KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI","abstract":"Real-time speech-to-speech (S2S) models excel at generating natural, low-latency conversational responses but often lack deep knowledge and semantic understanding. Conversely, cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior knowledge representation at the cost of high latency, which disrupts the flow of natural interaction. This paper introduces a novel hybrid architecture that bridges the gap between these two paradigms. Our framework processes user speech through an S2S transformer for immediate responsiveness while concurrently relaying the query to a powerful back-end LLM. The LLM's text-based response is then injected in real time to guide the S2S model's speech generation, effectively infusing its output with rich knowledge without the full latency penalty of a cascaded system. We evaluated our method using a speech-synthesized variant of the MT-Bench benchmark that consists of multi-turn question-answering sessions. The results demonstrate that our system substantially outperforms a baseline S2S model in response correctness, approaching that of a cascaded system, while maintaining a latency on par with the baseline.","short_abstract":"Real-time speech-to-speech (S2S) models excel at generating natural, low-latency conversational responses but often lack deep knowledge and semantic understanding. Conversely, cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior kn...","url_abs":"https://arxiv.org/abs/2510.02327","url_pdf":"https://arxiv.org/pdf/2510.02327v2","authors":"[\"So Kuroki\",\"Yotaro Kubo\",\"Takuya Akiba\",\"Yujin Tang\"]","published":"2025-09-26T00:46:34Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\",\"eess.AS\"]","methods":"[\"Transformer\",\"Large Language Model\",\"Language Model\"]","has_code":false}