具即時表情之三維虛擬分身重建系統應用於擴增實境遠程會議

Abstract

近年來，擴增實境 (AR) 引起了廣泛的關注。其中一種應用是用於擴增實境遠程會議。在這種以臨場體驗為重的應用中，我們期望能夠看到逼真的虛擬分身，並能在互動的過程中觀察到彼此的面部表情變化。然而，目前絕大多數的多使用者擴增實境系統是使用具靜態面部表情之虛擬分身，或是具預設面部動畫之虛擬分身來代表每個使用者。在電腦視覺領域中有許多研究提出基於深度學習的方法從單張影像中重建三維人臉模型，但目前尚未出現關於如何從三維人臉模型產生全身虛擬分身，以及如何將其實際運用到實時 AR/VR 應用中的相關研究。 在這篇論文中，我們提出了一個完整的系統架構來生成逼真的虛擬分身，並以最小的延遲持續更新虛擬分身的面部表情。首先，我們採用在當前最新的人臉重建技術，DECA，根據單張使用者的面部影像，初始化並重建出逼真的頭部模型。透過整合基於深度學習的性別分類器，系統將建立三維身體模型，以呈現更好的虛擬分身視覺效果。接著將頭部模型與身體模型進行合併來產生使用者的全身虛擬分身。在進行擴增實境遠程會議的過程中，我們利用筆記型電腦上的攝像頭來捕捉使用者的即時面部影像，並實時更新虛擬分身的面部表情及姿勢。透過此方法，使用者就能即時地觀察到彼此的面部表情變化。 我們在擴增實境遠程會議的應用場景中實現了即時虛擬分身重建系統。結果顯示本系統能呈現良好的視覺效果。另外，我們招募了7位受試者來體驗AR遠程會議中不同的虛擬分身，並以問卷調查的方式對系統進行評估。結果顯示我們的系統能有效的增進擴增實境協作系統中的臨場感並提供更好的使用者體驗。

Augmented reality (AR) technologies have attracted a lot of attention in recent years. One of its application is for AR teleconferencing. In this kind of multi-user AR systems where social presence plays an important role, we prefer to see realistic 3D avatars and observe the facial expression of the other remote users during the interaction. However, in most of the existing systems, the avatars are represented either by predefined 3D virtual characters with static face, or dynamic face with simple predefined animations. Many deep-learning based methods are proposed to reconstruct 3D face from single RGB image, but they do not investigate how to integrate the reconstructed face into full-body avatars, and how to efficiently apply the model into real-time AR/VR applications. In this thesis, we present a complete framework to generate realistic avatars of users, and efficiently update the facial expression of the avatars with minimized latency. We adopt a state-of-the-art face reconstruction model, DECA, to first reconstruct the realistic head model from a profile picture. To enhance the appearance of the avatars, we integrate a gender classifier module to generate the body mesh. The head model will then be merged together with the body model to provide a full-body avatar representation. During the AR teleconference, we update the facial expression and pose of users through images captured from webcam. This approach allows users to be aware of each other’s facial expression in real-time. To validate the effectiveness of our system, we conduct a user study to evaluate different types of avatar representations in AR teleconference scenario. The result shows that our approach significantly improves the social presence in the AR collaboration and the real-time avatar update framework can run in 10 fps.