{"ID":2856766,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.10580","arxiv_id":"2510.10580","title":"LQRS: Learned Query Re-optimization Framework for Spark SQL","abstract":"The query optimizer is a fundamental component of database management systems. Recent studies have shown that learned query optimizers outperform traditional cost-based query optimizers. However, they fail to exploit valuable runtime observations generated during query execution to dynamically re-optimize the plan, thereby limiting further improvements in query performance. To address this issue, we propose learned query re-optimization, which allows optimization decisions to be deferred to execution time and guided by actual runtime observations. We realize this idea through LQRS, a learned query re-optimization framework that builds upon Spark SQL, exploiting runtime observations for dynamic plan refinement. Specifically, LQRS employs a curriculum reinforcement learning strategy and jointly supports pre-execution and in-execution optimization, allowing knowledge learned during execution to directly benefit pre-execution planning. Furthermore, we design a plug-and-play planner extension built upon the extensibility interfaces of Spark SQL, enabling online plan modification. Experiments on Spark SQL demonstrate that LQRS reduces end-to-end execution time by up to 90% compared to other learned query optimizers and query re-optimization methods.","short_abstract":"The query optimizer is a fundamental component of database management systems. Recent studies have shown that learned query optimizers outperform traditional cost-based query optimizers. However, they fail to exploit valuable runtime observations generated during query execution to dynamically re-optimize the plan, the...","url_abs":"https://arxiv.org/abs/2510.10580","url_pdf":"https://arxiv.org/pdf/2510.10580v3","authors":"[\"Jiahao He\",\"Yutao Cui\",\"Cuiping Li\",\"Jikang Jiang\",\"Yuheng Hou\",\"Hong Chen\"]","published":"2025-10-12T12:55:32Z","proceeding":"cs.DB","tasks":"[\"cs.DB\"]","methods":"[\"Reinforcement Learning\"]","has_code":false}