{"ID":6537558,"CreatedAt":"2026-07-14T02:54:43.516908796Z","UpdatedAt":"2026-07-15T03:28:55.185153975Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.11415","arxiv_id":"2607.11415","title":"Stage-Level Executor Allocation in Apache Spark with Cost-Performance Trade-offs","abstract":"Allocating executors (i.e. compute resources) to distributed processing systems must balance resource costs of scaling-out unnecessarily against artificial, performance-limiting bottlenecks. Naive approaches may allocate executors at the application level, which have predictable costs and performance but are almost guaranteed to be sub-optimal for each of the thousands of diverse, individual stages executed by the application. Users may also have explicit preferences, such as completing an application within a specific time budget while minimizing cost, that existing solutions usually fail to support. We propose a novel method for determining the number of executors per stage in a serverless Apache Spark environment, enabling users to specify their desired cost-performance tradeoff. Our approach trains tree-ensemble models to estimate the run times and costs of a stage as a function of allocated resources. These estimates are then used to recommend resources for each stage individually. We evaluate our approach on TPC-DS and SQLStorm benchmarks and compare it against two baselines. Depending on the user-defined trade-off parameter and setup, our approach achieves approx. 50% cost savings across 103 TPC-DS queries with only a approx. 16% slowdown, and approx. 40.5% on 96 SQLStorm queries at a approx. 29% slowdown.","short_abstract":"Allocating executors (i.e. compute resources) to distributed processing systems must balance resource costs of scaling-out unnecessarily against artificial, performance-limiting bottlenecks. Naive approaches may allocate executors at the application level, which have predictable costs and performance but are almost gua...","url_abs":"https://arxiv.org/abs/2607.11415","url_pdf":"https://arxiv.org/pdf/2607.11415v1","authors":"[\"Miriam Rateike\",\"Isaac Waweru Wambugu\",\"Celia Cintas\",\"Michael Kaufmann\",\"Ioana Giurgiu\",\"Skyler Speakman\"]","published":"2026-07-13T11:23:04Z","proceeding":"cs.DC","tasks":"[\"cs.DC\"]","methods":"[]","has_code":false}