{"ID":2874719,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.04398","arxiv_id":"2509.04398","title":"IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation","abstract":"Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https://github.com/valeoai/peft-ipa .","short_abstract":"Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during...","url_abs":"https://arxiv.org/abs/2509.04398","url_pdf":"https://arxiv.org/pdf/2509.04398v3","authors":"[\"Yuan Yin\",\"Shashanka Venkataramanan\",\"Tuan-Hung Vu\",\"Andrei Bursuc\",\"Matthieu Cord\"]","published":"2025-09-04T17:10:01Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"LoRA\"]","has_code":false,"code_links":[{"ID":610162,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2874719,"paper_url":"https://arxiv.org/abs/2509.04398","paper_title":"IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation","repo_url":"https://github.com/valeoai/peft-ipa","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}