{"ID":2870127,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.22677","arxiv_id":"2509.22677","title":"Profit over Proxies: A Scalable Bayesian Decision Framework for Optimizing Multi-Variant Online Experiments","abstract":"Online controlled experiments (A/B tests) are fundamental to data-driven decision-making in the digital economy. However, their real-world application is frequently compromised by two critical shortcomings: the use of statistically flawed heuristics like \"p-value peeking\", which inflates false positive rates, and an over-reliance on proxy metrics like conversion rates, which can lead to decisions that inadvertently harm core business profitability. This paper addresses these challenges by introducing a comprehensive and scalable Bayesian decision framework designed for profit optimization in multi-variant (A/B/n) experiments. We propose a hierarchical Bayesian model that simultaneously estimates the probability of conversion (using a Beta-Bernoulli model) and the monetary value of that conversion (using a robust Bayesian model for the mean transaction value). Building on this, we employ a decision-theoretic stopping rule based on Expected Loss, enabling experiments to be concluded not only when a superior variant is identified but also when it becomes clear that no variant offers a practically significant improvement (stopping for futility). The framework successfully navigates \"revenue traps\" where a variant with a higher conversion rate would have resulted in a net financial loss, correctly terminates futile experiments early to conserve resources, and maintains strict statistical integrity throughout the monitoring process. Ultimately, this work provides a practical and principled methodology for organizations to move beyond simple A/B testing towards a mature, profit-driven experimentation culture, ensuring that statistical conclusions translate directly to strategic business value.","short_abstract":"Online controlled experiments (A/B tests) are fundamental to data-driven decision-making in the digital economy. However, their real-world application is frequently compromised by two critical shortcomings: the use of statistically flawed heuristics like \"p-value peeking\", which inflates false positive rates, and an ov...","url_abs":"https://arxiv.org/abs/2509.22677","url_pdf":"https://arxiv.org/pdf/2509.22677v1","authors":"[\"Srijesh Pillai\",\"Rajesh Kumar Chandrawat\"]","published":"2025-09-16T02:24:20Z","proceeding":"stat.AP","tasks":"[\"stat.AP\",\"cs.LG\",\"stat.ML\"]","methods":"[\"Generative Adversarial Network\"]","has_code":false}