{"ID":2825437,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.21041","arxiv_id":"2512.21041","title":"When LLMs fall short in Deductive Coding: Model Comparison and Human AI Collaboration Workflow Design","abstract":"With generative artificial intelligence driving the growth of dialogic data in education, automated coding is a promising direction for learning analytics to improve efficiency. This surge highlights the need to understand the nuances of student-AI interactions, especially those rare yet crucial. However, automated coding may struggle to capture these rare codes due to imbalanced data, while human coding remains time-consuming and labour-intensive. The current study examined the potential of large language models (LLMs) to approximate or replace humans in deductive, theory-driven coding, while also exploring how human-AI collaboration might support such coding tasks at scale. We compared the coding performance of small transformer classifiers (e.g., BERT) and LLMs in two datasets, with particular attention to imbalanced head-tail distributions in dialogue codes. Our results showed that LLMs did not outperform BERT-based models and exhibited systematic errors and biases in deductive coding tasks. We designed and evaluated a human-AI collaborative workflow that improved coding efficiency while maintaining coding reliability. Our findings reveal both the limitations of LLMs -- especially their difficulties with semantic similarity and theoretical interpretations and the indispensable role of human judgment -- while demonstrating the practical promise of human-AI collaborative workflows for coding.","short_abstract":"With generative artificial intelligence driving the growth of dialogic data in education, automated coding is a promising direction for learning analytics to improve efficiency. This surge highlights the need to understand the nuances of student-AI interactions, especially those rare yet crucial. However, automated cod...","url_abs":"https://arxiv.org/abs/2512.21041","url_pdf":"https://arxiv.org/pdf/2512.21041v1","authors":"[\"Zijian Li\",\"Luzhen Tang\",\"Mengyu Xia\",\"Xinyu Li\",\"Naping Chen\",\"Dragan Gašević\",\"Yizhou Fan\"]","published":"2025-12-24T08:10:02Z","proceeding":"cs.HC","tasks":"[\"cs.HC\"]","methods":"[\"Transformer\",\"Large Language Model\",\"Language Model\"]","has_code":false}