基於拉普拉斯金字塔的深度人臉影像光影重建

Abstract

這篇論文提出了一個人臉的光影重建系統，將參考的人臉影像之打光模式重建到目標人臉影像上。我們提出了一個深度學習的架構，並使用有兩個輸入分支的U-Net模組，使我們可以同時對參考人臉影像以及目標人臉影像進行學習。我們的學習目標不定在色彩空間上，而是讓模組學習人臉的拉普拉斯金字塔。假設金字塔的高頻段保留了更多材質資訊，我們將學習目標定在金字塔的低頻部分，並保留原始的高頻部分。實驗結果顯示這樣的設計可以更好的適應不同色光並保留更多細節。我們亦將人臉的深度影像以及人臉的面部剖析分割圖與人臉影像串聯做為輸入，提供給模組更多的人臉幾何資訊。 我們進行了讀者研究來評估我們的系統。在研究中，我們要求使用者指出重建光影後的影像其光線來向。此研究中我們達到了82.4%的正確辨識率。

This thesis presents a human face relighting system, which relights the target human face image in a specific lighting pattern according to the reference face image. We propose a 2-Input-Stream U-Net structure to learn from the target face and reference face simultaneously. Instead of learning the face on color space, we operate on frequency space by learning the Laplacian Pyramid of the face. Under the assumption that the high-frequency part preserves more information of textures, we only learn the low-frequency part and the high-frequency part remains the same. The experiment result shows that the model is more adaptive to lighting color and preserves more details. We also concatenate the depth map and face parsing segmentation map to the input stack to provide facial geometry information. To evaluate our model, we conduct a user study by asking users to identify the lighting direction of the relit image. We obtained 82.4% accuracy on this survey.