{"ID":2899540,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.00543","arxiv_id":"2507.00543","title":"Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications","abstract":"Despite growing interest in using large language models (LLMs) to automate annotation, their effectiveness in complex, nuanced, and multi-dimensional labelling tasks remains relatively underexplored. This study focuses on annotation for the search clarification task, leveraging a high-quality, multi-dimensional dataset that includes five distinct fine-grained annotation subtasks. Although LLMs have shown impressive capabilities in general settings, our study reveals that even state-of-the-art models struggle to replicate human-level performance in subjective or fine-grained evaluation tasks. Through a systematic assessment, we demonstrate that LLM predictions are often inconsistent, poorly calibrated, and highly sensitive to prompt variations. To address these limitations, we propose a simple yet effective human-in-the-loop (HITL) workflow that uses confidence thresholds and inter-model disagreement to selectively involve human review. Our findings show that this lightweight intervention significantly improves annotation reliability while reducing human effort by up to 45%, offering a relatively scalable and cost-effective yet accurate path forward for deploying LLMs in real-world evaluation settings.","short_abstract":"Despite growing interest in using large language models (LLMs) to automate annotation, their effectiveness in complex, nuanced, and multi-dimensional labelling tasks remains relatively underexplored. This study focuses on annotation for the search clarification task, leveraging a high-quality, multi-dimensional dataset...","url_abs":"https://arxiv.org/abs/2507.00543","url_pdf":"https://arxiv.org/pdf/2507.00543v1","authors":"[\"Leila Tavakoli\",\"Hamed Zamani\"]","published":"2025-07-01T08:04:58Z","proceeding":"cs.IR","tasks":"[\"cs.IR\",\"cs.HC\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}