{"ID":2896841,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.05610","arxiv_id":"2507.05610","title":"On the Inherent Privacy of Zeroth Order Projected Gradient Descent","abstract":"Differentially private zeroth-order optimization methods have recently gained popularity in private fine tuning of machine learning models due to their reduced memory requirements. Current approaches for privatizing zeroth-order methods rely on adding Gaussian noise to the estimated zeroth-order gradients. However, since the search direction in the zeroth-order methods is inherently random, researchers including Tang et al. (2024) and Zhang et al. (2024a) have raised an important question: is the inherent noise in zeroth-order estimators sufficient to ensure the overall differential privacy of the algorithm? This work settles this question for a class of oracle-based optimization algorithms where the oracle returns zeroth-order gradient estimates. In particular, we show that for a fixed initialization, there exist strongly convex objective functions such that running (Projected) Zeroth-Order Gradient Descent (ZO-GD) is not differentially private. Furthermore, we show that even with random initialization and without revealing (initial and) intermediate iterates, the privacy loss in ZO-GD can grow superlinearly with the number of iterations when minimizing convex objective functions.","short_abstract":"Differentially private zeroth-order optimization methods have recently gained popularity in private fine tuning of machine learning models due to their reduced memory requirements. Current approaches for privatizing zeroth-order methods rely on adding Gaussian noise to the estimated zeroth-order gradients. However, sin...","url_abs":"https://arxiv.org/abs/2507.05610","url_pdf":"https://arxiv.org/pdf/2507.05610v2","authors":"[\"Devansh Gupta\",\"Meisam Razaviyayn\",\"Vatsal Sharan\"]","published":"2025-07-08T02:38:14Z","proceeding":"math.OC","tasks":"[\"math.OC\",\"cs.LG\",\"stat.ML\"]","methods":"[]","has_code":false}