圖片與文字的多模態檢索

Abstract

多模態學習是近年來的一個挑戰，它將過去的單模態學習擴展到多模態，像是文本、圖像或語音。這種擴展需要模型去處理和理解來自多種模態的信息。在信息檢索中，傳統的檢索任務側重於單模態文檔和搜尋詞之間的相似性，而圖像-文本的檢索則假設大多數文本只有包含圖像的資訊。這種區分忽略了現實世界的關鍵字可能涉及文本內容、圖像概念或兩者皆包含。為了解決這個問題，我們介紹了基於文本和圖像的多模態檢索（Mr. Right），它包含了新穎且強大的多模態檢索數據集。我們使用具有豐富文本和圖像的維基百科數據集，並生成三種類型的關鍵字：文本相關、圖像相關和混合。為了驗證我們數據集的有效性，我們提供了一種多模態訓練方式並評估了以前文本檢索和圖像檢索的模型。結果表明，提出的多模態檢索可以提高檢索性能。然而，創建同時具有文本和圖像的多模態資訊仍然是一個挑戰。我們希望 Mr. Right 能讓我們更好地拓寬當前的檢索系統，並為信息檢索中多模態學習的進步做出貢獻。

Multimodal learning is a recent challenge that extends unimodal learning by generalizing its domain to diverse modalities, such as texts, images, or speech. This extension requires models to process and relate information from multiple modalities. In Information Retrieval, traditional retrieval tasks focus on the similarity between unimodal documents and queries, while image-text retrieval hypothesizes that most texts contain the scene context from images. This separation has ignored that real-world queries may involve text content, image captions, or both. To address this, we introduce Multimodal Retrieval on Representation of ImaGe witH Text (Mr. Right), a novel and robust dataset for multimodal retrieval. We utilize the Wikipedia dataset with rich text-image examples and generate three types of queries: text-related, image-related, and mixed. To validate the effectiveness of our dataset, we provide a multimodal training paradigm and evaluate previous text retrieval and image retrieval frameworks. The results show that proposed multimodal retrieval can improve retrieval performance, but creating a well-unified document representation with texts and images is still a challenge. We hope Mr. Right allows us to broaden current retrieval systems better and contributes to accelerating the advancement of multimodal learning in the Information Retrieval.