透過對比式學習建構潛在風格空間之任意圖像風格轉換模型

Abstract

我們在本研究之中提出了一個新穎的影像風格轉換方法，稱為「對比式估計自適應正規化」（Adaptive Normalization with Contrastive Estimation, AdaNCE）。它整合了編碼器—解碼器架構、對比式學習、生成對抗網路和循環一致性，結合實現在多元場景下的穩定且高品質的藝術風格轉換。在此模型架構中，我們會首先從預訓練好的 VGG19 網路提取出影像多層特徵，通過 AdaIN 的機制內容與風格統計量進行調配達到風格轉換的效果。此外，提取出的 AdaIN 風格統計量也可以經過由全連接層組成的類神經網路投影至 2048 維度的向量，以此結合對比式學習的概念來構建風格潛在空間。這使得不同風格之間可以有效地被區隔開，並保證相同風格的聚合度。

接下來，我們為了檢驗此論文中所提架構的效能，因而進一步地在 WikiArt、COCO、PBN 以及 DTD 等多個資料集上來實施了感知與量化實驗，實驗的衡量標準包括風格損失（style loss）、內容損失（content loss）、運算效能和視覺品質等項。得到的結果也印證本方法在萃取和保持風格特徵方面的穩定性以及其泛化能力。我們所推出的 AdaNCE 利用了對比式學習來做主要的模型優化與隱風格空間的建構，搭配對抗式網路、循環一致性的輔助，達到了速度與表現穩定度的理想平衡點。除此之外，我們也在相同實驗下通過消融實驗去驗證此模型每一個模塊的能力，此消融實驗證實了其三個組成部分—AdaIN、GAN 及對比式學習—對最終結果極為關鍵，只要缺乏任何之一的模塊，所達到的風格轉換效果即會因此而變差。

此外，我們還設計了使用者實驗，當中邀請來自國立臺灣大學的 77 位大學生進行主觀美感評估，包含辨識合成影像與真實藝術創作之間的真實性，以及在多模型比較情境下對合成品質的偏好。從中所得到的實驗結果揭示了：AdaNCE不僅能在多數的輸入中產生逼真的風格轉換作品，也兼具適度的運算速度與廣泛的風格泛化能力。綜合以上觀點所述，本研究提出之 AdaNCE 方法在維持內容結構與表現藝術風格細節方面裡展現了顯著的優勢，具有應用於多種風格轉換任務的潛力。

In this work, a brand new image style transfer approach is proposed: Adaptive Normalization with Contrastive Estimation (AdaNCE). For the purpose of stable and high-quality artistic style transfer across diverse scenarios, it integrates the encoder-decoder architecture, contrastive learning, generative adversarial networks, and cycle consistency. The overall process begins with the extraction of multi-layer features from the pre-trained VGG19 network. Through AdaIN, adjustments are made to the content and style statistics to achieve style transfer. The extracted AdaIN style statistics can further be projected into a 2048-dimensional vector using a fully connected neural network. This projection enables the construction of a latent style space, allowing contrastive learning to effectively distinguish different styles while ensuring cohesion within similar styles.

Several datasets—WikiArt, COCO, PBN, and DTD—were involved in the experiments to verify the effectiveness and generalization of the proposed method. These experiments comprehensively investigated style loss, content loss, computational efficiency, and visual quality. The final results show the robustness of the proposed method in extracting and preserving style features. Our AdaNCE balances the trade-offs between speed and stability, aided by the incorporation of GANs and cycle consistency. Ablation experiments proved that AdaIN, GAN, and contrastive learning were indispensable to the final results, since removing any of them would significantly degrade the performance.

Furthermore, a user study was conducted with 77 university students from National Taiwan University. Participants performed subjective aesthetic evaluations, including distinguishing between synthesized images and real artworks, as well as comparing synthesized quality preferences across different models. The final experimental results show that AdaNCE not only produces realistic style transfer outputs in most cases but also achieves moderate computational speed and broad style generalization capabilities.

In a nutshell, AdaNCE demonstrates significant advantages in maintaining content structure while expressing detailed artistic styles. It has the potential to be applied to various style transfer tasks.