基於RGB影像之場景理解定位系統應用於擴增實境遠程會議

Abstract

近年來，擴增實境開始被廣泛應用在各種場域，如倉儲、展演及教育等地。在相關應用的開發基礎中，能僅靠RGB影像就同時做到理解空間資訊及追蹤使用者姿態的定位演算法更是尤為重要。然而，目前市面上的相關套件普遍存在著一些限制，如Google公司開發的ARCore因為硬體需求，僅能使用在經過官方認證的設備上。這對於想要在自己的設備上進行擴增實境應用開發的一般使用者，造成了相當大的不便與阻礙。 在這篇論文中，我們提出了一個基於深度學習網路及SLAM的場景理解定位系統，可以僅靠RGB影像就同時做到場景理解及使用者定位。使用機器學習的方法，推導出使用者所處環境中有何種物件及其相對姿態，並透過SLAM匹配連續RGB影像幀的ORB特徵點來計算出使用者在空間中的位置及姿態。為了驗證此系統的有效性，我們使用3D引擎Unity作為開發平台，將此系統應用於我們設計的AR遠程會議應用中。使用者可透過配置於擴增實境眼鏡上的RGB攝像頭獲得周圍的環境資訊及使用者姿態，進而在鏡片上投影出其它會議參與者的虛擬分身在真實環境中的適當位置。 在實驗的部分，我們對系統中的各個模組進行了多種質性及量化分析，並邀請了數名受試者，請他們嘗試使用此AR遠端會議程式以評斷場景理解定位系統的好壞。由實驗結果及受試者的回饋問卷得知，此系統的定位穩定性及準確率獲得大多數使用者的肯定。我們期望我們所提出的場景理解定位系統能夠為更多獨立開發擴增實境應用的使用者，提供堅實且穩定的基礎技術。

Augmented reality has been widely used in various fields, such as warehousing, performances, and education. Among the technological developments in related applications, the positioning algorithm that can understand spatial information and track the user's posture at the same time using only RGB images is critical. However, the related development kits currently available on the market generally have limitations. For example, ARCore developed by Google can only be used on officially certified devices due to hardware requirements. This makes it impossible for developers who would like to work on augmented reality applications on their own devices to use those kits. In this thesis, we propose a scene understanding and positioning system based on neural networks and SLAM (Simultaneous Localization and Mapping), which can achieve scene understanding and user localization simultaneously using only RGB images. The machine learning method is first used to obtain the information of the class and relative pose of objects in the user's environment, and then the user's position and pose in space are calculated by matching the feature points on the continuous RGB image frames through the SLAM technique. In order to verify the effectiveness of our system, we use the 3D engine, called Unity, as the development platform, and implement our system for an AR teleconference application we proposed. The user can obtain the surrounding environment information and posture of the object (say, chair) of interest with the RGB camera on the augmented reality glasses, and then project the virtual avatar of another party of the conference at an appropriate location (say, chair) in the natural environment. In the experiment part, besides demonstrating the effectiveness and efficiency of our developed system, we also invite several subjects and ask them to try our AR teleconference application for user study, to collect users’ feelings about the quality of the scene understanding and positioning system. According to the feedback provided by the subjects, most users agree that our system does provide a real-time, stable, and accurate positioning system. We hope that our proposed scene understanding and positioning system provides a solid framework for more people to develop their augmented reality applications on different devices more easily.