基於深度學習應用於虛擬實境與物體互動之手部姿勢估計

Abstract

手部姿勢估計為電腦視覺中的一門熱門研究領域，其目的為從一張具有手部的影像中預測手上節點的位置建立手的姿勢，並由於虛擬實境、擴增實境及混合實境等科技的發展，近年來有為數不少相關的技術如火如荼的發展；然而儘管如此，因為其本身的困難性，例如在影像中，手可能會被自身所遮蔽、或在與外界物體互動的過程中被物體遮蔽等問題，相關的研究尚未能稱之為成熟，因此至今仍不斷有科學家投入該研究領域。 該論文的目的即想要發展一個能從深度影像上擷取資訊、並在3D空間中精確的預測手部姿勢的系統，以提升在虛擬實境等相關應用的使用者體驗。在本文中，我們提出一個利用資料數據來訓練一個深度學習的手部姿勢估計模型，並且是一個無論在手與物體互動與否都能進行估計的模型。文中運用的技術為卷積神經網路，並在網路中設計兩個不同的單元來提升該模型的訓練效果，分別稱為骨架差異層及物體操作層，前者考慮手的物體限制，後者加入手和物體互動之資訊；經由兩者的共同運作，該卷積神經網路所學習出來手部姿勢估計模型效果有明顯的提升。 在實驗中，我們將比較不同條件下所學習出來之手部姿勢估計模型的效果，並展示我們的方法儘管在非常具有挑戰性的環境下也能維持優異的效果，也證明了該篇論文提出來的方法能在手部姿勢估計上加入手的物理限制及物體互動等資訓來提升卷積神經網路的學習效果，並在手部姿勢估計的應用上如虛擬實境等應用上能得到更好、更自然的使用者互動體驗。

Hand Pose Estimation aims to predict the position of joints on a hand from an image, and it gets popular because of the emergence of VR/AR/MR technology. For example, we can integrate a depth sensor with a Head-Mounting Device (HMD) so that we can rebuild the hand in the VR world, and we can naturally communicate with the computer through this way. However, when we try to achieve this goal, since a hand tends to cause self-occlusion or external occlusion easily as it interacts with external objects, this problem becomes quite challenging. As a result, there are a lot of researches dedicating to this field for the better solutions of this problem. The purpose of this thesis is to develop a system that accurately estimate a hand pose in 3D space using depth images for VR applications. We propose a data-driven approach of training a deep learning model for hand pose estimation with object interaction. In the convolutional neural network (CNN) training procedure, we design a so-called skeleton-difference loss function, which can effectively learn the physical constraints of a hand, and also propose an object-manipulating loss function, which considers the knowledge of the hand-object interaction, to enhance the performance. In the experiments we have conducted for the hand pose estimation under different conditions, the results validate the robustness and the performance of our system, and show that our method is able to predict the joints more accurately in those challenging environmental settings. Such appealing results may be attributed to consideration of physical joint relationship as well as object information, which in turn can be applied to future VR/AR/MR systems for more natural experience.