{"ID":2843691,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.06635","arxiv_id":"2511.06635","title":"Can LLM Annotations Replace User Clicks for Learning to Rank?","abstract":"Large-scale supervised data is essential for training modern ranking models, but obtaining high-quality human annotations is costly. Click data has been widely used as a low-cost alternative, and with recent advances in large language models (LLMs), LLM-based relevance annotation has emerged as another promising annotation. This paper investigates whether LLM annotations can replace click data for learning to rank (LTR) by conducting a comprehensive comparison across multiple dimensions. Experiments on both a public dataset, TianGong-ST, and an industrial dataset, Baidu-Click, show that click-supervised models perform better on high-frequency queries, while LLM annotation-supervised models are more effective on medium- and low-frequency queries. Further analysis shows that click-supervised models are better at capturing document-level signals such as authority or quality, while LLM annotation-supervised models are more effective at modeling semantic matching between queries and documents and at distinguishing relevant from non-relevant documents. Motivated by these observations, we explore two training strategies -- data scheduling and frequency-aware multi-objective learning -- that integrate both supervision signals. Both approaches enhance ranking performance across queries at all frequency levels, with the latter being more effective. Our code is available at https://github.com/Trustworthy-Information-Access/LLMAnn_Click.","short_abstract":"Large-scale supervised data is essential for training modern ranking models, but obtaining high-quality human annotations is costly. Click data has been widely used as a low-cost alternative, and with recent advances in large language models (LLMs), LLM-based relevance annotation has emerged as another promising annota...","url_abs":"https://arxiv.org/abs/2511.06635","url_pdf":"https://arxiv.org/pdf/2511.06635v1","authors":"[\"Lulu Yu\",\"Keping Bi\",\"Jiafeng Guo\",\"Shihao Liu\",\"Shuaiqiang Wang\",\"Dawei Yin\",\"Xueqi Cheng\"]","published":"2025-11-10T02:26:14Z","proceeding":"cs.IR","tasks":"[\"cs.IR\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":607243,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2843691,"paper_url":"https://arxiv.org/abs/2511.06635","paper_title":"Can LLM Annotations Replace User Clicks for Learning to Rank?","repo_url":"https://github.com/Trustworthy-Information-Access/LLMAnn_Click","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}