{"ID":2895197,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.09566","arxiv_id":"2507.09566","title":"Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems","abstract":"A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key advantage of this approach is its ability to simultaneously serve multiple test groups, each with distinct offline performance metrics, in an online experiment controlled by a single model. The method is model-agnostic for systems with a neural network backbone, enabling broad applicability across architectures and domains. We validate the strategy through a large-scale online experiment in the field of session-based recommender systems on the OTTO e-commerce platform. The online experiment identifies significant alignments between offline metrics and real-word click-through rate, post-click conversion rate and units sold. Our strategy provides industry practitioners with a valuable tool for understanding offline-to-online metric relationships and making informed, data-driven decisions.","short_abstract":"A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key ad...","url_abs":"https://arxiv.org/abs/2507.09566","url_pdf":"https://arxiv.org/pdf/2507.09566v1","authors":"[\"Timo Wilm\",\"Philipp Normann\"]","published":"2025-07-13T10:24:41Z","proceeding":"cs.IR","tasks":"[\"cs.IR\",\"cs.AI\",\"cs.LG\"]","methods":"[]","has_code":false}