以神經網路整合雙串流軌跡資料用於計算任意人類動作次數

Abstract

雖然深度學習已經在電腦視覺方面取得相當地勝利，辨識人類的動作重複次數仍是十分具挑戰性的問題，在病人進行復健以及人們進行重量訓練等時刻皆需要計算動作重複的次數。解決此問題的困難點在於重複動作時動作間的細微差異、人們在畫面中重複動作時相機的拍攝視角移動以及對應不同重複動作需要不同的處理。為了解決這個問題，我們收集了一個新的資料集以及建立了一個嶄新的網路架構，人類重複動作計算網路，用以計算任意的人類重複動作。我們的網路使用人們隨著時間進行重複動作之軌跡的頻域資訊來計算人類重複動作的次數，實驗結果指出我們的網路在計算人類任意動作的重複次數方面有優於以往的表現。另外由於人類進行重複動作的影片取得較為困難，我們也製作了一個資料集，使用產生波形的方式來模擬人類重複動作時的軌跡，並使用此資料集預訓練我們的網路以獲得更精確的計算能力。人類重複動作計算網路並不僅僅只能計算人類重複動作的次數，在實驗中，以物體進行重複動作時的軌跡輸入網路也可以獲得物體重複動作的次數。

Although deep neural network has achieved great success in computer vision recently, the problem of determining repetitions of arbitrary periodic human actions is still challenging. The difficulties lay in varying frame length of repetitions, temporal localization of human beings and different features corresponding to different motions. Moreover, the demand of human action repetition counting is rising in medical rehabilitation and sport events, etc. To address this problem, we construct a human action dataset and propose a brand new framework, Human Action Repetition Counter (HARC), which could work on arbitrary human actions with a single architecture. Our HARC learns to count repetitions of human action in the time-frequency domain determined after few pilot studies. The experiments show that HARC outperforms previous counting methods on benchmarks. Additionally, we design novel learning strategies by generating effective synthetic data to pretrain our network, which can further boost the performance and reach more accurate results. We also demonstrate that our HARC is also capable of counting the periodic object motions. Our dataset, YT_Human_Segments dataset, will be publicly available which will benefit future researches.