{"ID":2836221,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.21038","arxiv_id":"2511.21038","title":"Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels","abstract":"Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \\emph{natural} demonstrations (with correct labels) and \\emph{inverted} demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl.","short_abstract":"Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \\emph{natural} demonstrations (with correct labels) and \\emph{inverted} demonstrations...","url_abs":"https://arxiv.org/abs/2511.21038","url_pdf":"https://arxiv.org/pdf/2511.21038v1","authors":"[\"Anantha Padmanaban Krishna Kumar\"]","published":"2025-11-26T04:14:33Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\",\"cs.LG\"]","methods":"[\"Large Language Model\"]","has_code":false,"code_links":[{"ID":606581,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2836221,"paper_url":"https://arxiv.org/abs/2511.21038","paper_title":"Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels","repo_url":"https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}