{"ID":2843109,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.07790","arxiv_id":"2511.07790","title":"CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis","abstract":"Sentiments about the reproducibility of cited papers in downstream literature offer community perspectives and have shown as a promising signal of the actual reproducibility of published findings. To train effective models to effectively predict reproducibility-oriented sentiments and further systematically study their correlation with reproducibility, we introduce the CC30k dataset, comprising a total of 30,734 citation contexts in machine learning papers. Each citation context is labeled with one of three reproducibility-oriented sentiment labels: Positive, Negative, or Neutral, reflecting the cited paper's perceived reproducibility or replicability. Of these, 25,829 are labeled through crowdsourcing, supplemented with negatives generated through a controlled pipeline to counter the scarcity of negative labels. Unlike traditional sentiment analysis datasets, CC30k focuses on reproducibility-oriented sentiments, addressing a research gap in resources for computational reproducibility studies. The dataset was created through a pipeline that includes robust data cleansing, careful crowd selection, and thorough validation. The resulting dataset achieves a labeling accuracy of 94%. We then demonstrated that the performance of three large language models significantly improves on the reproducibility-oriented sentiment classification after fine-tuning using our dataset. The dataset lays the foundation for large-scale assessments of the reproducibility of machine learning papers. The CC30k dataset and the Jupyter notebooks used to produce and analyze the dataset are publicly available at https://github.com/lamps-lab/CC30k .","short_abstract":"Sentiments about the reproducibility of cited papers in downstream literature offer community perspectives and have shown as a promising signal of the actual reproducibility of published findings. To train effective models to effectively predict reproducibility-oriented sentiments and further systematically study their...","url_abs":"https://arxiv.org/abs/2511.07790","url_pdf":"https://arxiv.org/pdf/2511.07790v1","authors":"[\"Rochana R. Obadage\",\"Sarah M. Rajtmajer\",\"Jian Wu\"]","published":"2025-11-11T03:13:17Z","proceeding":"cs.DL","tasks":"[\"cs.DL\",\"cs.CL\"]","methods":"[\"Language Model\"]","has_code":false,"code_links":[{"ID":607183,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2843109,"paper_url":"https://arxiv.org/abs/2511.07790","paper_title":"CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis","repo_url":"https://github.com/lamps-lab/CC30k","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}