DAVSE: 基於擴散模型的生成式影像結合語音增強方法

Abstract

近年來，語音視覺語音增強（Audio-Visual Speech Enhancement, AVSE）因其能夠在嘈雜環境中提升語音可懂度與品質，受到廣泛關注。儘管去噪效能已有顯著進展，AVSE 系統仍面臨兩項主要挑戰：（1）判別式方法可能引入不自然的語音失真，抵消降噪帶來的效益；（2）視覺訊號的整合往往伴隨額外的運算成本。

本論文提出一種基於擴散模型的創新方法，旨在解決上述挑戰。我們的系統採用一個基於分數的擴散模型來學習乾淨語音資料的先驗分佈。透過這一先驗知識，系統能從偏離學習分佈的嘈雜或混響輸入中推斷出乾淨語音。此外，音訊與視覺輸入透過交叉注意力模組整合至條件噪聲分數網路中，並未增加額外的計算成本。

實驗結果顯示，所提出的 DAVSE 系統在提升語音品質與減少生成性瑕疵（如語音混淆）方面，相較於僅使用音訊的語音增強系統有明顯優勢。此外，實驗也證實交叉注意力模組能有效地融合音訊與視覺資訊。

In recent years, audio-visual speech enhancement (AVSE) has attracted considerable attention for its ability to improve speech intelligibility and quality in noisy environments. Despite advances in denoising performance, two major challenges remain in AVSE systems: (1) discriminative approaches can introduce unpleasant speech distortions that may negate the benefits of noise reduction, and (2) integrating visual input often leads to increased processing costs.

This thesis presents a novel diffusion model-based approach to address these challenges. Our system utilizes a score-based diffusion model to learn the prior distribution of clean speech data. This prior knowledge enables the system to infer clean speech from noisy or reverberant input signals that deviate from the learned distribution. In addition, audio and visual inputs are integrated into the noise conditional score network through cross-attention modules, without incurring additional computational costs.

Experimental evaluations demonstrate that the proposed DAVSE system significantly improves speech quality and reduces generative artifacts, such as phonetic confusions, compared to audio-only SE systems. Furthermore, the results confirm the effectiveness of cross-attention modules in seamlessly incorporating audio and visual information.