{"ID":2834126,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2601.02370","arxiv_id":"2601.02370","title":"Variance-Aware LLM Annotation for Strategy Research: Sources, Diagnostics, and a Protocol for Reliable Measurement","abstract":"Large language models (LLMs) offer strategy researchers powerful tools for annotating text at scale, but treating LLM-generated labels as deterministic overlooks substantial instability. Grounded in content analysis and generalizability theory, we diagnose five variance sources: construct specification, interface effects, model preferences, output extraction, and system-level aggregation. Empirical demonstrations show that minor design choices-prompt phrasing, model selection-can shift outcomes by 12-85 percentage points. Such variance threatens not only reproducibility but econometric identification: annotation errors correlated with covariates bias parameter estimates regardless of average accuracy. We develop a variance-aware protocol specifying sampling budgets, aggregation rules, and reporting standards, and delineate scope conditions where LLM annotation should not be used. These contributions transform LLM-based annotation from ad hoc practice into auditable measurement infrastructure.","short_abstract":"Large language models (LLMs) offer strategy researchers powerful tools for annotating text at scale, but treating LLM-generated labels as deterministic overlooks substantial instability. Grounded in content analysis and generalizability theory, we diagnose five variance sources: construct specification, interface effec...","url_abs":"https://arxiv.org/abs/2601.02370","url_pdf":"https://arxiv.org/pdf/2601.02370v3","authors":"[\"Arnaldo Camuffo\",\"Alfonso Gambardella\",\"Saeid Kazemi\",\"Jakub Malachowski\",\"Abhinav Pandey\"]","published":"2025-12-02T18:02:20Z","proceeding":"cs.CY","tasks":"[\"cs.CY\",\"cs.CL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}