雙流膨脹卷積網路之台灣手語單字辨識

Abstract

在台灣手語辨識的相關研究中，往往都是透過輔助裝置獲得影片中的手部、運動軌跡等資訊並利用傳統特徵擷取算法來實驗與闡述，使得這些研究往往僅能解決在特定情境、特定資料集下的辨識問題，且許多論文並未明確指示出實驗資料集的數量與分布。本論文首先建立了一個針對台灣手語單字影片的資料集，共有4位不同人在不同背景比出常用之63種手語單字，每種單字種類有8至9個影片，共有538支影片；接著透過近年在影像動作辨識上取得良好效果的雙流膨脹卷積網路以泛化的形式解決手語辨識問題，在測試集上達到87.3%的準確率，並透過實驗及錯誤分析給出針對光流和rgb frame作為輸入的往後改進方向。最後將訓練完的模型應用到能即時展示辨識結果的程式中。

In the related research on Taiwanese sign language recognition, the hand position and motion trajectory information in the sign language video are often obtained through auxiliary devices, and traditional feature extraction algorithms like SIFT, SURF are used for experiment. These make the related research only applicable to problems for a specific data set or scenario. Furthermore, many papers do not clearly indicate the experimental data set like quantity and distribution. In this thesis, we first established a data set for Taiwanese sign language vocabulary videos. A total of 4 different people gestures featuring the 63 commonly used sign language vocabularies in different backgrounds were considered. Each vocabulary has 8 to 9 videos and a total of 538 videos. Secondly, we apply two-stream dilated convolutional network, which has achieved good results in video motion recognition as reported in the literature, to solve the Taiwanese sign language recognition problem in a generalized form. In the experiment, we reach an accuracy of 87.3% on the test set, and through error analysis, we indicate future improvement on rgb frame and optical flow as inputs for model. Finally, the trained model is applied to a program that can display the recognition results in real time.