{"ID":2895000,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.10536","arxiv_id":"2507.10536","title":"On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance","abstract":"In this work, we analyze the optimization behaviour of common private learning optimization algorithms under heavy-tail class imbalanced distribution. We show that, in a stylized model, optimizing with Gradient Descent with differential privacy (DP-GD) suffers when learning low-frequency classes, whereas optimization algorithms that estimate second-order information do not. In particular, DP-AdamBC that removes the DP bias from estimating loss curvature is a crucial component to avoid the ill-condition caused by heavy-tail class imbalance, and empirically fits the data better with $\\approx8\\%$ and $\\approx5\\%$ increase in training accuracy when learning the least frequent classes on both controlled experiments and real data respectively.","short_abstract":"In this work, we analyze the optimization behaviour of common private learning optimization algorithms under heavy-tail class imbalanced distribution. We show that, in a stylized model, optimizing with Gradient Descent with differential privacy (DP-GD) suffers when learning low-frequency classes, whereas optimization a...","url_abs":"https://arxiv.org/abs/2507.10536","url_pdf":"https://arxiv.org/pdf/2507.10536v1","authors":"[\"Qiaoyue Tang\",\"Alain Zhiyanov\",\"Mathias Lécuyer\"]","published":"2025-07-14T17:57:08Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[]","has_code":false}