{"ID":2874743,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.04442","arxiv_id":"2509.04442","title":"Delta Activations: A Representation for Finetuned Large Language Models","abstract":"The success of powerful open source Large Language Models (LLMs) has enabled the community to create a vast collection of post-trained models adapted to specific tasks and domains. However, navigating and understanding these models remains challenging due to inconsistent metadata and unstructured repositories. We introduce Delta Activations, a method to represent finetuned models as vector embeddings by measuring shifts in their internal activations relative to a base model. This representation allows for effective clustering by domain and task, revealing structure in the model landscape. Delta Activations also demonstrate desirable properties: it is robust across finetuning settings and exhibits an additive property when finetuning datasets are mixed. In addition, we show that Delta Activations can embed tasks via few-shot finetuning, and further explore its use for model selection and merging. We hope Delta Activations can facilitate the practice of reusing publicly available models. Code is available at https://github.com/OscarXZQ/delta_activations.","short_abstract":"The success of powerful open source Large Language Models (LLMs) has enabled the community to create a vast collection of post-trained models adapted to specific tasks and domains. However, navigating and understanding these models remains challenging due to inconsistent metadata and unstructured repositories. We intro...","url_abs":"https://arxiv.org/abs/2509.04442","url_pdf":"https://arxiv.org/pdf/2509.04442v1","authors":"[\"Zhiqiu Xu\",\"Amish Sethi\",\"Mayur Naik\",\"Ser-Nam Lim\"]","published":"2025-09-04T17:59:06Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.CL\",\"cs.IR\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":610165,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2874743,"paper_url":"https://arxiv.org/abs/2509.04442","paper_title":"Delta Activations: A Representation for Finetuned Large Language Models","repo_url":"https://github.com/OscarXZQ/delta_activations","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}