{"ID":2878284,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.00078","arxiv_id":"2509.00078","title":"ChipChat: Low-Latency Cascaded Conversational Agent in MLX","abstract":"The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device voice agents remains an open question. While end-to-end approaches promise theoretical advantages, cascaded systems (CSs) continue to outperform them in language understanding tasks, despite being constrained by sequential processing latency. In this work, we introduce ChipChat, a novel low-latency CS that overcomes traditional bottlenecks through architectural innovations and streaming optimizations. Our system integrates streaming (a) conversational speech recognition with mixture-of-experts, (b) state-action augmented LLM, (c) text-to-speech synthesis, (d) neural vocoder, and (e) speaker modeling. Implemented using MLX, ChipChat achieves sub-second response latency on a Mac Studio without dedicated GPUs, while preserving user privacy through complete on-device processing. Our work shows that strategically redesigned CSs can overcome their historical latency limitations, offering a promising path forward for practical voice-based AI agents.","short_abstract":"The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device voice agents remains an open question. While end-to-end approaches promise theoretical advantages, cascaded systems (CSs) continue to outperform them in language understanding tasks,...","url_abs":"https://arxiv.org/abs/2509.00078","url_pdf":"https://arxiv.org/pdf/2509.00078v1","authors":"[\"Tatiana Likhomanenko\",\"Luke Carlson\",\"Richard He Bai\",\"Zijin Gu\",\"Han Tran\",\"Zakaria Aldeneh\",\"Yizhe Zhang\",\"Ruixiang Zhang\",\"Huangjie Zheng\",\"Navdeep Jaitly\"]","published":"2025-08-26T20:40:24Z","proceeding":"eess.AS","tasks":"[\"eess.AS\",\"cs.CL\",\"cs.LG\",\"cs.SD\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}