視覺音訊和相似網路用於影片重複動作計數

Abstract

我們提出 VASsNet 作為視頻中重複動作計數的一種新穎方法，它結合了視覺、音頻及其相似度矩陣。在之前的工作中，視覺、視覺相似度和音頻特徵已分別用於重複運動計數。然而，由於缺乏對這三個方面信息的有效整合，在模糊和/或包含快速運動的視頻中很難獲得良好的計數結果。VASsNet 由四個路徑構成，即視覺、視覺相似性、音頻和音頻相似性路徑。通過採用多層跨模態信息融合方法，通過橫向連接有效地集成從這些路徑中提取的信息。通過實驗，我們演示瞭如何利用相似矩陣路徑來解決視頻中短期運動引起的先前無法檢測到的重複動作計數的問題；以及音頻路徑如何幫助提高模糊視頻的計數準確性。實驗結果表明，VASsNet 在 Countix 和Countix-AV 數據集上實現了最先進的性能。

We propose VASsNet as a novel approach for repetitive action counting in video, which incorporates Vision, Audio, as well as their Similarity matrices. In previous works, vision, vision similarity, and audio features have been separately used for repetitive motion counting. However, due to the lack of effective integration of information from these three aspects, it is difficult to achieve decent counting results in videos which are blurry and/or contain rapid movements. The VASsNet is structured with four pathways, namely the vision, vision similarity, audio and audio similarity pathways. The information extracted from these pathways is effectively integrated through lateral connections by employing a multi-layers cross-modal information fusion approach. Through experiments, we demonstrate how the similarity matrix pathways can be utilized to solve the problem of the previously undetectable repetitive action counting which is caused by short-term motion in videos; and how the audio pathway can help to enhance the counting accuracy with blurry videos. Experiment results show that VASsNet achieves the state-of-the-art performance on Countix and Countix-AV datasets.