{"ID":5346720,"CreatedAt":"2026-06-30T04:09:55.830587294Z","UpdatedAt":"2026-07-02T14:44:57.46949413Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.30391","arxiv_id":"2606.30391","title":"Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs","abstract":"As LLM inference becomes a major cloud workload, its growing energy footprint makes cluster-wide energy optimization increasingly important. Serverless LLM serving helps platforms absorb traffic volatility by elastically sharing GPU resources across models, but this sharing also makes energy optimization difficult. Multiple co-resident models run under one device-wide operating point, while their resource demands and latency slack change across execution phases and load conditions. As a result, minimizing energy requires coordinated scheduling across request placement, runtime resource adaptation, and workload consolidation. We present Festina, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving. Unlike common global-local schedulers that focus on throughput or tail latency, Festina makes energy-first decisions by jointly coordinating request placement, SM partitioning, and GPU operating points under TTFT/TBT SLOs. In our system, a lightweight global scheduler performs fast, SLO-safe, energy-aware placement using constant-time lookups from offline profiles and GPU state summaries. On each GPU, a phase-aware local scheduler continuously adapts task batching and compute resources to minimize power consumption. Festina further performs energy-aware workload consolidation to reduce GPUs' static power consumption via SLO-aware migration. Comparison with four SOTA LLM serving systems and one DVFS-augmented system demonstrates that Festina reduces energy consumption by up to 56% while maintaining parity in SLO attainment (within a 2% margin)","short_abstract":"As LLM inference becomes a major cloud workload, its growing energy footprint makes cluster-wide energy optimization increasingly important. Serverless LLM serving helps platforms absorb traffic volatility by elastically sharing GPU resources across models, but this sharing also makes energy optimization difficult. Mul...","url_abs":"https://arxiv.org/abs/2606.30391","url_pdf":"https://arxiv.org/pdf/2606.30391v1","authors":"[\"Tianyu Wang\",\"Gourav Rattihalli\",\"Aditya Dhakal\",\"Longfei Shangguan\",\"Dejan Milojicic\"]","published":"2026-06-29T14:44:24Z","proceeding":"cs.DC","tasks":"[\"cs.DC\"]","methods":"[\"Large Language Model\"]","has_code":false}