{"ID":2863530,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.24655","arxiv_id":"2509.24655","title":"HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling","abstract":"Language models are increasingly applied to biological sequences like proteins and mRNA, yet their default Euclidean geometry may mismatch the hierarchical structures inherent to biological data. While hyperbolic geometry provides a better alternative for accommodating hierarchical data, it has yet to find a way into language modeling for mRNA sequences. In this work, we introduce HyperHELM, a framework that implements masked language model pre-training in hyperbolic space for mRNA sequences. Using a hybrid design with hyperbolic layers atop Euclidean backbone, HyperHELM aligns learned representations with the biological hierarchy defined by the relationship between mRNA and amino acids. Across multiple multi-species datasets, it outperforms Euclidean baselines on 9 out of 10 tasks involving property prediction, with 10% improvement on average, and excels in out-of-distribution generalization to long and low-GC content sequences; for antibody region annotation, it surpasses hierarchy-aware Euclidean models by 3% in annotation accuracy. Our results highlight hyperbolic geometry as an effective inductive bias for hierarchical language modeling of mRNA sequences.","short_abstract":"Language models are increasingly applied to biological sequences like proteins and mRNA, yet their default Euclidean geometry may mismatch the hierarchical structures inherent to biological data. While hyperbolic geometry provides a better alternative for accommodating hierarchical data, it has yet to find a way into l...","url_abs":"https://arxiv.org/abs/2509.24655","url_pdf":"https://arxiv.org/pdf/2509.24655v2","authors":"[\"Max van Spengler\",\"Artem Moskalev\",\"Tommaso Mansi\",\"Mangal Prakash\",\"Rui Liao\"]","published":"2025-09-29T12:04:15Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"q-bio.GN\"]","methods":"[\"Language Model\"]","has_code":false}