{"ID":2864320,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.23861","arxiv_id":"2509.23861","title":"Investigating Multi-layer Representations for Dense Passage Retrieval","abstract":"Dense retrieval models usually adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model usually contain different kinds of linguistic knowledge, and behave differently during fine-tuning. Therefore, we propose to investigate utilizing representations from multiple encoder layers to make up the representation of a document, which we denote Multi-layer Representations (MLR). We first investigate how representations in different layers affect MLR's performance under the multi-vector retrieval setting, and then propose to leverage pooling strategies to reduce multi-vector models to single-vector ones to improve retrieval efficiency. Experiments demonstrate the effectiveness of MLR over dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting, as well as demonstrate that it works well with other advanced training techniques such as retrieval-oriented pre-training and hard negative mining.","short_abstract":"Dense retrieval models usually adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model usually contain different kinds of linguistic knowledge, and behave differently during fine-t...","url_abs":"https://arxiv.org/abs/2509.23861","url_pdf":"https://arxiv.org/pdf/2509.23861v1","authors":"[\"Zhongbin Xie\",\"Thomas Lukasiewicz\"]","published":"2025-09-28T13:00:53Z","proceeding":"cs.IR","tasks":"[\"cs.IR\"]","methods":"[\"Language Model\"]","has_code":false}