基於SMPL-X應用擴散模型的文字到可動畫三維人體虛擬分身服裝生成

Abstract

近年來，隨著元宇宙(Metaverse)技術的發展，虛擬分身（Avatar）扮演著至關重要的角色。但傳統的三維虛擬分身建模需要花費大量的時間及資源。為了解決這個問題，設計一個可以直接從文字產生三維虛擬分身的系統變得越來越重要，這種方法不僅降低了分身創建的門檻，還提供了更大的彈性。透過分析使用者提供的文字敘述，如性別、人種、身形與服裝等資訊，系統可以幫助其他模組預測及生成出符合敘述的虛擬分身。

比起傳統的角色建模，由文字生成虛擬分身雖然節省了大量時間，但現有方法還是需要數小時來生成，難以快速地讓使用者得到想要的結果。因此，本論文提出了TeCHAvatar，一個基於SMPL-X的虛擬分身生成系統，分別預測人種，形狀和生成紋理圖，並整合成一個完整的虛擬分身。這個過程僅需花費數十秒。由於現有的SOTA方法所產生出的大多是屬於隱式的虛擬分身，難以建立一組帶有骨骼的三維虛擬人物網格，無法應對後續像是骨骼動畫的應用。因此我們以SMPL-X的骨骼為基礎，將服裝綁定在同一骨骼上，並在虛擬分身變形後自動調整關節點到適當的位置，確保骨骼還是有效的。此外，為了提升系統對服裝提示詞的理解程度，我們提出了CTLoRA，用來幫助生成指定服裝材質的紋理圖。我們還設計了專用的著色器，用於渲染虛擬分身。

In recent years, with the development of metaverse technology, avatars have played a crucial role. However, traditional 3D avatar modeling requires a significant amount of time and resources. To address this issue, designing a system that can generate 3D avatars directly from text has become increasingly important. This approach not only lowers the barrier to avatar creation but also provides greater flexibility. By analyzing user-provided textual descriptions, such as gender, race, body shape, and clothing information, the system can assist other modules in predicting and generating avatars that match the descriptions.

Compared to traditional character modeling, generating avatars from text saves a lot of time, but the existing methods still require several hours to produce results, making it difficult for users to quickly obtain the desired outcome. Therefore, this thesis proposes TeCHAvatar, an avatar generation system based on SMPL-X, which separately predicts race, shape, and generates texture maps from the given text, integrating them into a complete avatar. This process takes only a few seconds. Since most of the avatars produced by current state-of-the-art methods are implicit and cannot establish a 3D avatar mesh with skeletons, they are not suitable for subsequent applications such as skeleton animation. Hence, we use the skeleton of SMPL-X, binding the clothing to the same skeleton and automatically adjusting the joints to appropriate positions after the avatar deforms, ensuring the skeleton remains effective. Additionally, to enhance the system's understanding of clothing prompts, we propose CTLoRA, which helps generate texture maps for specified clothing materials. We also design specialized shaders for rendering the final avatars.