{"ID":6267747,"CreatedAt":"2026-07-10T01:11:38.759438437Z","UpdatedAt":"2026-07-14T10:01:43.020260556Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.07862","arxiv_id":"2607.07862","title":"CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems","abstract":"The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTA-pipelining reduces latency by up to 31.8% compared to micro-batching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary.","short_abstract":"The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has...","url_abs":"https://arxiv.org/abs/2607.07862","url_pdf":"https://arxiv.org/pdf/2607.07862v1","authors":"[\"Tingkai Liu\",\"Muralidhar Andoorveedu\",\"Sanjoy Das\",\"Sanjay Patel\",\"Volodymyr Kindratenko\"]","published":"2026-07-08T18:54:08Z","proceeding":"cs.DC","tasks":"[\"cs.DC\",\"cs.LG\"]","methods":"[\"Language Model\"]","has_code":false}