{"ID":2879197,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.17126","arxiv_id":"2508.17126","title":"Token Homogenization under Positional Bias","abstract":"This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise similarity analysis and controlled experiments, we demonstrate that tokens systematically lose distinctiveness during processing, particularly when biased toward extremal positions. Our findings confirm both the existence of homogenization and its dependence on positional attention mechanisms.","short_abstract":"This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise sim...","url_abs":"https://arxiv.org/abs/2508.17126","url_pdf":"https://arxiv.org/pdf/2508.17126v1","authors":"[\"Viacheslav Yusupov\",\"Danil Maksimov\",\"Ameliia Alaeva\",\"Tatiana Zaitceva\",\"Antipina Anna\",\"Anna Vasileva\",\"Chenlin Liu\",\"Rayuth Chheng\",\"Danil Sazanakov\",\"Andrey Chetvergov\",\"Alina Ermilova\",\"Egor Shvetsov\"]","published":"2025-08-23T19:59:05Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\",\"cs.LG\"]","methods":"[\"Transformer\",\"Language Model\"]","has_code":false}