{"ID":2852367,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.18828","arxiv_id":"2510.18828","title":"Actor-Free Continuous Control via Structurally Maximizable Q-Functions","abstract":"Value-based algorithms are a cornerstone of off-policy reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely on estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the Q-value over the entire action space becomes computationally infeasible. To address this, actor-critic methods are typically employed, where a critic is trained on off-policy data to estimate Q-values, and an actor is trained to maximize the critic's output. Despite their popularity, these methods often suffer from instability during training. In this work, we propose a purely value-based framework for continuous control that revisits structural maximization of Q-functions, introducing a set of key architectural and algorithmic choices to enable efficient and stable learning. We evaluate the proposed actor-free Q-learning approach on a range of standard simulation tasks, demonstrating performance and sample efficiency on par with state-of-the-art baselines, without the cost of learning a separate actor. Particularly, in environments with constrained action spaces, where the value functions are typically non-smooth, our method with structural maximization outperforms traditional actor-critic methods with gradient-based maximization. We have released our code at https://github.com/USC-Lira/Q3C.","short_abstract":"Value-based algorithms are a cornerstone of off-policy reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely on estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the...","url_abs":"https://arxiv.org/abs/2510.18828","url_pdf":"https://arxiv.org/pdf/2510.18828v1","authors":"[\"Yigit Korkmaz\",\"Urvi Bhuwania\",\"Ayush Jain\",\"Erdem Bıyık\"]","published":"2025-10-21T17:24:27Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.RO\",\"stat.ML\"]","methods":"[\"Reinforcement Learning\"]","has_code":false,"code_links":[{"ID":607991,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2852367,"paper_url":"https://arxiv.org/abs/2510.18828","paper_title":"Actor-Free Continuous Control via Structurally Maximizable Q-Functions","repo_url":"https://github.com/USC-Lira/Q3C","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}