文本中心對齊的多模態監督學習

Abstract

本文探討了多模態監督學習中模態不匹配的挑戰，其指的是推理過程中出現的模態和訓練時出現的模態不同的情況，我們提出了一種創新方法TAMML(文本中心對齊的多模態監督學習)，該方法利用具有上下文學習能力的大語言模型和基礎模型提高多模態系統在這種情況下的泛化能力。通過利用文本作為統一語義空間的獨特特性，本文展示了在處理未見過、多樣、不可預測的模態組合時的顯著改進。所提出的解決方法不僅能夠適應不同的模態，還能保持穩健的性能，展示了基礎模型在克服傳統模型框架在嵌入表示的局限性的潛力。本研究通過提供一種靈活且有效的方法，為動態且不確定模態可用性的真實應用作出貢獻。

This paper addresses the challenge of modality mismatch in supervised learning, where the modalities available during inference differ from those available during training. We propose an innovative method, TAMML(Text-centric Alignment for Multi-Modality Supervised Learning), that utilizes Large Language Models with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, this paper demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. The proposed solution not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.