基於StyleGAN的高斯混合形變之三維風格化頭像建模

Abstract

3D Gaussian Blendshapes 的提出，使得從單目視頻中即時重建可驅動動畫的頭像成為可能。另一方面，基於 StyleGAN 框架的 Toonify 方法在人臉圖像風格化領域中亦展現出廣泛應用潛力。為實現風格化的三維高斯混合形變（Gaussian Blenshapes）頭像建模，我們提出一種高效的兩階段框架 ToonifyGB。在第一階段（風格化視頻生成），我們採用改良的 StyleGAN 模型對輸入視頻進行風格轉換，突破了需於固定解析度下裁剪與對齊人臉的限制，從而產生更穩定且一致的風格化視頻輸出。此穩定輸出有助於後續階段更準確地捕捉面部表情細節，進一步提升混合形變的建模效率與動畫品質。第二階段（基於高斯混合形變的三維風格化頭像合成）中，我們從風格化視頻中學習風格化頭像的一個中性模型以及一組表情混合形變。這些混合形變在保留真實人臉表情特徵的同時，結合中性模型，可實現多樣且動態的風格化頭像生成。我們在多個基準數據集上，針對兩種不同風格（奧術"Arcane"與皮克斯"Pixar"）進行實驗，結果證明所提出的方法在視覺一致性與表情驅動能力上均具備良好的通用性與表現力。

The introduction of 3D Gaussian blendshapes has enabled the real-time reconstruction of animatable head avatars from monocular video. Toonify, a StyleGAN-based method, is widely adopted for facial image stylization. To extend Toonify for synthesizing diverse stylized 3D head avatars using Gaussian blendshapes, we propose an efficient two-stage framework, ToonifyGB. In Stage 1 (stylized video generation), we adopt an improved StyleGAN to generate the stylized video from the input video frames, which overcomes the limitation of requiring fixed resolution, cropping aligned faces as preprocessing for normal StyleGAN. This process produces a more stable stylized video, which helps Gaussian blendshapes more accurately capture high-frequency facial details, which facilitates the synthesis of high-quality animations in the next stage. In Stage 2 (Gaussian blendshapes synthesis), we learn a stylized neutral head model and a set of expression blendshapes from the generated stylized video. By combining the neutral head model with expression blendshapes, ToonifyGB can efficiently render stylized avatars with arbitrary expressions. We validate the effectiveness of ToonifyGB on benchmark datasets using two representative styles: Arcane and Pixar.