透過解耦學習影片問答中的時空間關係

Abstract

雖然最近大規模的影片語言預訓練在影片問答方面有了很大的進展，但空間建模的設計沒有圖像語言模型的那麼精緻；現有的時間建模方式也受到模態之間沒有對齊的影響。為了學習精緻的視覺理解，我們將時空建模解耦，並提出了一種結合圖像和影片語言編碼器的混合結構。前者獨立於時間從較大但稀疏採樣的 影格中理解空間語義，而後者在較低空間但較高時間解析度下捕捉時間動態。另外，為了幫助影片語言模型學習影片問答的時間關係，我們提出了一種新穎的預訓練目標，即時間引用建模，它要求模型辨別影片序列中事件的時間位置。透過廣泛且詳細的實驗，我們證明這個方法做得比以前在數量級更大的資料集上預訓練的研究更好。

While recent large­-scale video-­language pre-­training made great progress in video question answering, the design of spatial modeling is less fine­-grained than that of image­ language models; existing practices of temporal modeling also suffer from weak and noisy alignment between modalities. To learn fine-grained visual understanding, we decou­ple spatial­-temporal modeling and propose a hybrid pipeline integrating an image­ and a video-­language encoder. The former encodes spatial semantics from larger but sparsely sampled frames independently of time, while the latter models temporal dynamics at lower spatial but higher temporal resolution. To help the video­-language model learn temporal relations for video QA, we propose a novel pre-­training objective, Temporal Referring Modeling, which requires the model to identify temporal positions of events in video se­quences. Extensive and detailed experiments demonstrate that our model outperforms previous work that pre-trained on orders of magnitude larger datasets.