任務式多模態對話狀態追蹤的遷移學習能力之研究

Abstract

在多模態任務導向型對話中，代理人（agent）需要理解使用者感受到的多模態環境的能力。Simmc和Simmc2是針對購物場景的多模態任務導向對話的資料集。Simmc在傢俱領域使用了輪播的視角（carousel view），並在時尚領域使用了在真實世界使用者視角下的圖片作為多模態上下文。與Simmc不同，Simmc2使用了逼真的場景圖片。由於環境設置的難度不同，相較於Simmc，Simmc2在資料收集上更加困難。在現實世界中，像Simmc2這樣情境的資料可能很少，而從像Simmc這樣的情境中收集資料則更為容易。

在本論文中，我們藉由在Simmc上進行預訓練（pre-training），然後在資源較少的Simmc2上進行微調（fine-tuning），來研究多模態任務導向對話的可轉移性。結果顯示，在Simmc上進行預訓練可以提高聯合準確性（joint accuracy），並且每個意圖（intent）都從預訓練中受益。

In multi-modal task-oriented dialogues, agents need the ability to comprehend the multi-modal context perceived by the user. For this purpose, Simmc and Simmc2 were introduced as multi-modal task-oriented dialogue datasets, specifically designed for shopping scenarios. Simmc incorporates a carousel view within the furniture domain and real-world images captured from the user's perspective in the fashion domain as their multi-modal contexts. In contrast, Simmc2 utilizes photo-realistic scenes, making the data collection process more challenging due to its elaborate setup compared to Simmc. In real-world scenarios, obtaining data similar to Simmc2 may be a rarity, whereas collecting data akin to Simmc is comparatively easier.

This thesis aims to explore the transferability of multi-modal task-oriented dialogue by employing a two-step process: pre-training on the Simmc dataset, followed by fine-tuning on the lower resource Simmc2 dataset. The results indicate that pre-training on Simmc significantly enhances joint accuracy, and all intents demonstrate improvements from the pre-training process.