使用多尺度注意模型之弱監督學習式影片動作片段重定位

Abstract

影片重定位的目標是根據我們所給予的查詢影片，在參考影片中找出我們所感興趣的片段-本研究探討的是動作片段。本論文提出多尺度注意力機制模型來處理弱監督學習下的影片重定位問題。在訓練模型的過程中，不需要使用參考影片中動作片段的位置資訊。整體模型由三個模組所組成：第一部份使用預訓練的C3D模型提取特徵；接著設計一個多尺度注意力模組，根據不同尺度的特徵來計算參考影片與查詢影片的相似度：這樣的機制可保留更多時間資訊。最後，定位模組對參考影片中每個時間點的特徵進行預測，判斷其是否屬於動作片段。本研究使用共注意力損失函數來訓練模型，利用特徵之間的相似度，將動作片段從參考影片中分離出來。此外，只要加上交叉熵損失函數，我們的模型可以輕易修改成監督式學習的版本。我們在一個公開的資料集上測試，無論是監督式學習或是弱監督式學習，都獲得目前最好的效果。

Video re-localization aims to localize the segment we are interested in, an activity segment in this work, in the reference according to the query video we given. In this thesis, we propose an attention-based model to deal with this problem under a weakly-supervised setting. In other words, we train our model without the label of the action clip location in the reference video. Our model is composed of three major modules. In the first module, we employ a pre-trained C3D network as a feature extractor. For the second module, we design an attention mechanism to compute the similarity between the query video and a reference video based on multiscaled features. It can better preserve the local temporal information. Finally, in the final module, a localization layer predicts whether the feature of each time step in the reference video belongs to the query action segment. The full model is based on co-attention loss which separates an action instance from the reference video by utilizing the similarity between features. Our model can also be easily revised to fully supervised setting by updating the cross entropy loss to exploit the given location information. We evaluate our model on a public dataset and achieve the state-of-the-art performance under both weakly supervised and fully supervised settings.