{"ID":6537738,"CreatedAt":"2026-07-14T02:54:43.516908796Z","UpdatedAt":"2026-07-15T03:28:55.185153975Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.11211","arxiv_id":"2607.11211","title":"FastTPS: An Optimized Method for LLM Token Phase for AI accelerators","abstract":"The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in decoder-only LLMs inference, the inherent low parallelism leads to reduced throughput and suboptimal utilization of the computing units on artificial intelligence (AI) accelerators, particularly when handling long-sequence inputs that impose significant memory overhead. Recently, many reported methods have been developed as potential solutions, since they emerge with numeric deviation. This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: (1) AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusion of Attention, (2) high-efficiency and high-accuracy 'RoPE' attention based on the tiling optimized FLAT, and (3) highly-fused MLP with fine-grain pipeline scheduling. Our results confirm that FastTPS significantly alleviates memory bottlenecks in the token phase, delivering a 6x speed improvement (compared to none-fusion) on an AMD Ryzen AI 300 series NPU with BF16 precision while sustaining 93% peak memory bandwidth utilization during Phi3-mini-4k-instruct inference.","short_abstract":"The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in decoder-only LLMs inference, the inherent low parallelism leads to reduced throughput and suboptimal utilization of the computing units on art...","url_abs":"https://arxiv.org/abs/2607.11211","url_pdf":"https://arxiv.org/pdf/2607.11211v1","authors":"[\"Wenzong Yang\",\"Danyang Zhang\",\"Kun Cao\",\"Tejus Siddagangaiah\",\"Rajeev Patwari\",\"Zhanxing Pu\",\"Siyin Kong\",\"Zijiang Yang\",\"Hao Zhu\",\"Varun Sharma\",\"Yue Gao\",\"Tianping Li\",\"Fan Yang\",\"Jicheng Chen\",\"Yushan Chen\",\"Fennian Zhao\",\"Aaron Ng\",\"Elliott Delaye\",\"Ashish Sirasao\",\"Sudip Nag\"]","published":"2026-07-13T08:02:26Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}