文本主題的向量表徵模型及其多模態任務應用

Abstract

近年來，圖像與文字資料間的跨模態訊息已經受到廣泛的研究及應用。不過，文本中的主題性資訊 (如文章主旨或論述的中心思想) 卻從未被應用至多模態任務中，這樣的資訊如何被機器理解與表徵也未曾被深入探討。有鑑於此，本論文提出「融合主題表徵」來作為文本主題在多模態任務中的向量表徵形式，並論證文本主題與文字、圖片等模態同樣能承載重要語意訊息。

本論文藉由兩種子表徵來建構融合主題表徵：透過BERTopic主題模型產出的sentence-BERT向量作為全域主題表徵，及透過node2vec和graphSAGE從主題標籤網路(hashtag network)所產生的節點向量作為局部主題表徵。接著，本論文設計三種不同的任務來檢驗融合主題表徵的效果：文本主題相似度評測任務主要比較人類與機器對文本主題概念的理解，而其餘兩項多模態預測任務 (貼文熱度預測及廣告辨識) 則透過置換不同模態組合來分析融合主題表徵是否能增進下游任務的表現。

研究結果顯示，當融合主題表徵被作為多模態文本表徵的一部分時，模型在下游任務的表現可以提昇約5%。這說明了文本主題能輔助其他模態的預測表現，並在多模態標表徵中攜帶有助於模型預測的主題訊息。此外，人類與機器在評斷文本主題相似度時的Spearman相關係數達到0.44，表示融合主題表徵大致能夠模擬人類認知中的文本主題概念。最後，融合主題表徵的兩項子表徵分別能擷取不同粒度的主題資訊，而兩者融合時彼此的資訊呈現互補的模式。

Recent developments in multimodal machine learning have made extensive explorations into the cross-modal relationships between textual and visual data. However, topical information in documents (such as central ideas and discoursive focus in texts) has never been implemented to multimodal tasks, and its vector representation still remain under-researched. In light of this, the present thesis proposes Integrated Topic Embeddings (ITEs) to represent document topics in multimodal prediction tasks, and argues that they serve as an equally informative modality as text and images.

This thesis combines two elements to form integrated topic embeddings: global topic embeddings, which are sentence-BERT embeddings generated from BERTopic, and local topic embeddings, which are node embeddings generated by node2vec and graphSAGE from a hashtag network. Three experiments are then designed to validate the effectiveness of ITEs: the topic similarity rating task aims to compare human cognition and machine understanding of document topics, and two ablation tasks (popularity prediction and advertisement detection) examine whether the machine predicts better with document topics fused in the multimodal document representation.

The results indicate that when incorporating ITEs, multimodal models can boost task performances by up to 5%. This demonstrates that document topics are able to support other modalities, and they serve as an informative component in multimodal document representations. In addition, topic information encoded in ITEs moderately resembles that of human perception, as inferred from an average Spearman's correlation of 0.44 between human and the machines's ratings of document topic similarity. Finally, qualitative assessments on ITEs imply that the hashtag network and BERTopic capture different layers and granularity of topical information, and the two are complementary when combined as ITEs.