利用品質感知的多層池化方法強化文本嵌入

Abstract

文本嵌入在各種自然語言處理任務中扮演關鍵角色。近期多項研究聚焦於以大型語言模型為基礎的文本嵌入方法，並在多種下游任務中展現出卓越的效能。然而，這些方法大多僅使用模型的最後一層來提取嵌入，而有研究指出，最後一層未必能提供最佳表示。為此，本研究提出一種結合多層資訊的品質感知池化方法，根據每一層的品質指標自動分配權重，以生成更具表現力的文本嵌入。實驗結果顯示，所提方法在多數下游任務中皆能提升性能，驗證其有效性與泛化能力。

Text embeddings play a crucial role in various natural language processing tasks. Recent studies have focused on using large language models (LLMs) to generate high-quality embeddings, achieving remarkable performance on a wide range of downstream tasks. However, most existing methods rely solely on the final layer of the model to extract embeddings, despite evidence suggesting that the last layer may not always provide the most informative representation. To address this, we propose a quality-aware multi-layer pooling approach that integrates information from all layers and assigns weights based on layer-wise quality scores. Experimental results demonstrate that our method consistently improves performance across multiple downstream tasks, validating its effectiveness and generalizability.