透過問題相關幀定位和空間引導改善影片理解

Abstract

影片問答是一項具有挑戰性的任務，需要模型在豐富的影片內容中準確地識別和理解相關信息。傳統方法試圖通過考慮視覺問題的關聯性，強調特定幀中的相關信息。然而，由於缺乏因果幀的真實標籤，這種關聯性只能透過「間接地」學習，導致了「聚焦錯誤」(misfocus)的問題。為了解決這個問題，我們提出了一種新的訓練框架，稱為「空間蒸餾與因果幀定位」(Spatial distillation And Reliable Causal frame localization)，利用現成的圖像問答模型來幫助影片問答模型更好地理解影片的時間和空間維度中的相關信息。具體而言，我們利用圖像問答模型的視覺問題和答案先驗知識來獲得因果幀的偽標籤，並在時間維度上「直接地」指導影片聚焦在相關聯的幀。此外，由於圖像模型具有出色的空間推理能力，我們通過知識蒸餾將這種知識轉移到影片模型中。我們的方法不依賴於特定模型，在各種基準測試中均優於以前的方法。此外，它在多個影片問答模型(包括預訓練和非預訓練模型)上都能穩定提升性能。

Video Question Answering (Video QA) is a challenging task that requires models to accurately identify and contextualize relevant information within abundant video contents. Conventional approaches attempt to emphasize related information in specific frames by considering the visual-question relationship. However, the absence of ground-truth of causal frames makes such a relationship can only be learned implicitly, leading to the "misfocus" issue. To address this, we propose a novel training pipeline called "Spatial distillation And Reliable Causal frame localization", which leverages an off-the-shelf image QA model to make the video QA model better grasp relevant information in temporal and spatial dimensions of the video. Specifically, we use the visual-question and answer priors from an image QA model to obtain pseudo ground-truth of causal frames and explicitly guide the video QA model in the temporal dimension. Moreover, due to the superior spatial reasoning ability of image models, we transfer such knowledge to video models via knowledge distillation. Our model-agnostic approach outperforms previous methods on various benchmarks. Besides, it consistently improves performance (up to 5%) across several video QA models, including pre-trained and non pre-trained models.