多模態檢索增強生成：文本引導的跨模態增強技術用於高效本地部署

Abstract

多模態檢索增強生成（RAG）系統在處理同時包含文本與視覺元素的文檔時，面臨維持跨模態一致性的重大挑戰。傳統方法受到模態隔離問題困擾，文本與圖像檢索各自獨立運作，經常導致回應結果結合了來自不同來源的無關文本與視覺內容。此問題在資源受限環境中更加嚴重，小型語言模型缺乏執行複雜跨模態增強任務所需的精密指令遵循能力。我們提出文本導向多模態 RAG，這是一種創新方法，利用文本檢索結果透過文本導向跨模態增強來改善圖像檢索效果。我們的方法透過首先執行文本檢索，接著使用 TF-IDF 分析萃取判別性關鍵詞以建立增強查詢進行圖像搜尋，解決模態隔離問題。我們引入新的重排序機制，優先選擇與檢索文本來自相同來源文檔的圖像，確保跨模態一致性。我們採用確定性關鍵詞萃取而非讓小型語言模型遵循複雜指令，避免了小型視覺語言模型無法處理複雜任務的限制。透過配對比較分析進行的全面實驗評估顯示在多項指標上獲得顯著改善。我們的方法在圖像檢索品質方面皆有提升（MRR：+20.1%，nDCG：+22.0%），同時維持適合本地部署情境的計算效率。比較分析顯示我們的文本導向方法優於小型語言模型輔助增強方法，展現出卓越的跨模態檢索效能。這些結果驗證了結構感知跨模態增強在資源受限多模態 RAG 部署上的有效性。

Multimodal Retrieval-Augmented Generation (RAG) systems face significant challenges in maintaining cross-modal coherence when processing documents containing both text and visual elements. Traditional approaches suffer from modal isolation, where text and image retrieval operate independently, often resulting in responses that combine unre-lated textual and visual content from disparate sources. This problem is exacerbated in resource-constrained environments where small language models (SLMs) lack the sophis-ticated instruction-following capabilities required for complex cross-modal enhancement tasks. We propose Text-Informed Multimodal RAG, a novel approach that leverages text retrieval results to enhance image retrieval through text-informed cross-modal enhance-ment. Our method addresses the modal isolation problem by first performing text retrieval, then extracting discriminative keywords using TF-IDF analysis to create enhanced queries for image search. We introduce new re-ranking mechanism that prioritizes images from the same source documents as retrieved text, ensuring cross-modal coherence. This ap-proach circumvents the limitations of small vision-language models by employing deter-ministic keyword extraction rather than complex instruction following. Comprehensive experimental evaluation through paired-comparison analysis demonstrates substantial im-provements across multiple metrics. Our approach achieves significant gains in image retrieval quality (MRR: +20.1%, nDCG: +22.0%), while maintaining computational eff-ciency suitable for local deployment scenarios. Comparative analysis reveals that our text-informed method substantially outperforms SLM-aided enhancement approaches, demon-strating superior cross-modal retrieval effectiveness. These results validate the effective-ness of structure-aware cross-modal enhancement for resource-constrained multimodal RAG deployment.