以多元特徵方法優化邊際設備上的影片描述

Abstract

機器生成影片描述能增進機器人與人類溝通的能力。而邊緣運算能提供更快的回應速度和保護隱私的節點運算。若能在邊緣設備上運行「機器生成影片描述」，將可以使智能機器人與人類進行即時互動，同時減少上傳原始資料到雲端伺服器所造成的隱私問題。然而，類神經網路所需的大量計算資源，會導致影片描述很難運行在邊際設備上。而先前的文獻也缺乏相關研究。本論文旨在解決影片描述在邊緣設備上的運算量問題。 為了找到現行的運算瓶頸，我們分析了影片描述模型的架構以及所需運算量。從分析結果來看，影片描述模型的運算瓶頸是卷積網路特徵提取。為了解決這個問題，我們提出了一種名為「 多模態槓桿」的方法。此方法使用較小的卷積網路特徵提取器，並透過其他低計算量特徵資料（如音頻特徵或其他描述資料）來補償預期的準確度損失。 我們的結果顯示，多模態槓桿可以在降低計算負荷 92％ 的同時，只讓 METEOR 分數下降 4％。而在邊緣設備所需的運算時間也減少了30％。

Video description is important for its ability to enable smart robots’ interaction with humans. At the same time, edge computing provides fast round-trip and privacy-preserving computing. Combining video description and edge computing decrease the response time of smart robots and reduces the privacy-concern of transmitting raw data to cloud servers. However, the heavy computational load makes running video description on edge difficult. Also, there is a lack of studies of such application in the literature. In this thesis, we investigate the computational problem of video description on edge devices. To find the computational bottleneck, we first analyze a CNN- based video description method. The bottleneck turns out to be the CNN feature extraction. To solve the problem, we utilize an approach called “multi-modal leveraging” which uses small CNN feature extractors and compensates for the expected accuracy loss by other low-cost features such as audio features or metadata. Our experiments showed that the proposed method reduced 30% of the inference time and 92% of the computation load with a 4% accuracy drop on METEOR scores.