離散化語音自監督模型特徵用於多語言語音辨識

Abstract

語音自監督學習模型在各種語音處理任務中展示了卓越的能力。使用語音自監督模型連續特徵訓練模型雖然性能強大，但卻受限於其高計算和存儲成本。另一方面，雖然使用語音自監督模型離散化特徵訓練模型的性能有所下降，卻可通過去重複化以及字節對編碼，大量降低了傳輸和存儲成本，並提高了輸入序列的訓練效率。為了提升使用語音自監督模型離散化特徵訓練自動語音識別模型中的性能，我們提出了一種新穎的融合機制，整合了兩種離散特徵。這種融合機制保留了離散特徵的所有優點，同時通過整合離散特徵的互補信息來增強模型的性能。此外，我們還探索了「自增強」離散特徵，它對單一連續特徵進行轉換，消除了融合機制對多個語音自監督學習模型的依賴，還進一步降低了推理成本。在包括LibriSpeech和ML-SUPERB在內的基準測試上的實驗結果顯示，與非融合的基準比較對象相比，我們提出的方法有高達19%和24%的字符錯誤率相對進步量，證明了我們方法的有效性。

Self-supervised learning (SSL) models have shown exceptional capabilities across various speech-processing tasks. Continuous SSL representations are effective but suffer from high computational and storage demands. On the other hand, discrete SSL representations, although with degraded performance, reduce transmission and storage costs, and improve input sequence efficiency through de-duplication and subword-modeling. To boost the performance of discrete representations for ASR, we introduce a novel fusion mechanism that integrates two discrete representations. The fusion mechanism preserves all the benefits of discrete representation while enhancing the model's performance by integrating complementary information. Additionally, we explore ''self-augmented'' discrete representations, which apply transformations to a single continuous SSL representation, eliminating the fusion mechanism's dependency on multiple SSL models and further decreasing its inference costs. Experimental results on benchmarks, including LibriSpeech and ML-SUPERB, indicate up to 19% and 24% relative character error rate improvement compared with the non-fusion baseline, validating the effectiveness of our proposed methods.