{"ID":2848747,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.25904","arxiv_id":"2510.25904","title":"Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation","abstract":"The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time.","short_abstract":"The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated dataset...","url_abs":"https://arxiv.org/abs/2510.25904","url_pdf":"https://arxiv.org/pdf/2510.25904v1","authors":"[\"Frederico Belcavello\",\"Ely Matos\",\"Arthur Lorenzi\",\"Lisandra Bonoto\",\"Lívia Ruiz\",\"Luiz Fernando Pereira\",\"Victor Herbst\",\"Yulla Navarro\",\"Helen de Andrade Abreu\",\"Lívia Dutra\",\"Tiago Timponi Torrent\"]","published":"2025-10-29T19:13:48Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Large Language Model\"]","has_code":false}