DiffusionNet:利用粒子擴散辨識連續動作的高效率雙流神經網路

Abstract

動作識別是電腦視覺一個非常熱門的研究領域，其應用廣泛，普遍影響著人們的生活，像是監視設備、人機互動系統，都仰賴從影片中辨識人體動作的能力。在一般單一的圖片影像辨識中，大多利用空間資訊來獲取訊息，然而，影片並不同於圖像處理簡單，影片辨識多了豐富卻也複雜的時間資訊。多數研究會善用雙流架構深度模型來分別處理空間與時間特徵，但如此龐大的模型常常難以實作在嵌入式系統中，反而降低實用性。此外，時間特徵的抽取也常常成為限制速度的瓶頸，要如何設計一個在速度上能達到即時性、準確度能達到實用性，卻又足夠輕量，使其能放在移動設備中，便成為一個很重要的問題。 這篇論文提出了一個名為DiffusionNet的輕量級雙流模型架構，其在動作識別任務上可以達到良好的準確率以及即時性。我們假設光流具有粒子的特性，會隨著時間擴散，而我們的實驗證實此立論基礎能幫助我們更有效地萃取時間特徵。另外，我們也設計了自動調節的Focal Loss損失函數、注意力集中機制來幫助模型更有效地獲得空間領域特徵。在輕量化模型的部分，我們透過MobileNetV2的Depth-wise和Point-wise Convolution計算結構來降低計算複雜度，透過實作CUDA版本Pyramidal Lucas-Kanade演算法來快速地生成光流，解決多數傳統光流法中，時間上無法即時性的瓶頸問題。

Human action recognition is one of the most active research fields in computer vision. Many applications, such as video surveillance systems, human-computer interaction, require recognizing human actions in video sequences. Different from image classification tasks, which only have spatial information, action recognition contains a lot of noisy temporal information. Many state-of-the-art methods are based on the two-stream architecture, which has two neural networks to process spatial and temporal data separately and fuse the results for final prediction. However, the size of temporal information is an order of magnitude larger than spatial information. Previous works based on traditional CNN models (\eg VGG and ResNet) cannot be deployed to edge devices. Moreover, most current models adopt TV-L1 or Brox algorithms to calculate optical flow, which has a long latency that prohibits real-time recognition. Those two issues limit the deployment of action recognition in timing-critical tasks such as autonomous navigation. In this thesis, we propose a new action recognition network architecture that can achieve real-time performance on edge devices. We employ the depthwise and pointwise convolutions for reducing computational complexity and leverage the Pyramidal Lucas-Kanade algorithm for shortening the latency. Besides, we propose an assumption about optical flow diffusion, and our experiments show that it helps model extracting temporal information. Moreover, we design the automated focal loss function and attention mechanism to extract features more efficiently. Empirical results show that our method can reduce much computational complexity while preserving similar accuracy as the original deep two-stream model. Therefore, our model can meet the strict requirements of running real-time applications on mobile devices.