個人化文字轉語音：針對新語者、口音及語言的快速輕量化少樣本適應

Abstract

在這篇博士論文中，我們提出了一項關於推進文字轉語音（TTS）系統實現有效個性化的全面研究，特別是在需要少量樣本學習和零樣本學習的情境中。核心重點在於開發TTS系統中語音克隆和跨語言適應的新方法，旨在解決有限數據下快速適應的迫切需求。 在第一章中，我們為研究奠定了基礎，強調了為個別語音和語言定制TTS系統的重要性。這為我進行少量樣本學習、語音克隆和跨語言TTS適應的探索奠定了基礎。我的目標是有效地使用最少的訓練樣本，解決將TTS系統適應新語音和新語言所面臨的挑戰。 第二章對相關文獻進行了全面的回顧，深入探討了TTS技術的演進。這包括對端到端神經TTS模型、語音合成技術、多語言TTS和各種適應方法的深入討論。值得注意的是，我們考察了TTS與元學習的交叉點，特別是在跨語言語音克隆的背景下，這為我大部分研究提供了基礎。 第三章詳細介紹了我的研究架構基礎，以Transformer模塊和FastSpeech 2為中心。我們探索了FastSpeech 2在多語音和多語言應用中的適應性以及對訓練期間未遇到的領域的微調。 在第四章中，我們介紹了Meta-TTS，這是一種創新的用於少量樣本語音適應TTS的元學習方法。該章深入探討了Meta-TTS的訓練方法、微調策略和一整套評估指標。通過廣泛的實驗，我們展示了Meta-TTS在使用有限數據進行語音克隆時達到高度語音相似性和自然性的有效性。 第五章轉向個性化、輕量級TTS，通過自適應結構化剪枝。我們介紹了自適應結構化剪枝，這是一種在微調TTS模型時提高參數效率和計算速度的方法。這項技術作為Meta-TTS的補充，專注於提高微調效率和模型性能，對於高效和有效的TTS系統至關重要。 在第六章中，我們探討了使用可轉移音素嵌入的少量樣本跨語言TTS，提出了一種超越現有轉移學習方法的新方法。該章包括嚴格的實驗評估，展示了音素嵌入轉移在語言適應任務中的有效性。 論文在第七章結束，我們總結了這項研究對TTS領域的重大貢獻。我們的工作不僅推進了語音克隆和跨語言適應的最新技術，而且還為個性化TTS應用開辟了新途徑，特別是在資源受限的設置中。

In this dissertation, we present a comprehensive study on advancing text-to-speech (TTS) systems towards effective personalization, especially in scenarios requiring few-shot and zero-shot learning. The core focus lies in developing novel approaches for voice cloning and cross-lingual adaptation in TTS systems, aiming to address the pressing need for rapid adaptability with limited data. In the initial chapter, we establish the groundwork for the study, emphasizing the significance of customizing TTS systems for individual voices and languages. This sets the stage for my exploration into few-shot learning, voice cloning, and cross-lingual TTS adaptation. My objective is to tackle the challenges associated with adapting TTS systems to new voices and languages efficiently, using minimal training samples. Chapter 2 provides a thorough review of the relevant literature, delving into the evolution of TTS technology. This includes an in-depth discussion of end-to-end neural TTS models, vocoder technologies, multilingual TTS, and various adaptation methods. Notably, we examine the intersection of TTS and meta-learning, particularly in the context of cross-lingual voice cloning, which forms the basis for much of my research. The third chapter details the architectural foundations of my research, centered around the Transformer block and FastSpeech 2. We explore the adaptations of FastSpeech 2 for multi-speaker and multi-lingual applications and its fine-tuning for domains not encountered during training. In Chapter 4, we introduce Meta-TTS, an innovative meta-learning approach for few-shot speaker adaptive TTS. This chapter delves into the Meta-TTS training methodology, fine-tuning strategies, and a comprehensive set of evaluation metrics. Through extensive experiments, we demonstrate the effectiveness of Meta-TTS in achieving high speaker similarity and naturalness in voice cloning with limited data. Chapter 5 shifts focus to personalized, lightweight TTS through adaptive structured pruning. We present adaptive structured pruning, a method that improves parameter efficiency and computational speed in fine-tuning TTS models. This technique, complementing Meta-TTS, focuses on enhancing fine-tuning efficiency and model performance, crucial for efficient and effective TTS systems. In Chapter 6, we explore few-shot cross-lingual TTS using transferable phoneme embedding, proposing a new methodology that surpasses existing transfer learning approaches. This chapter includes rigorous experimental evaluation, showcasing the effectiveness of phoneme embedding transfer in language adaptation tasks. The dissertation concludes with Chapter 7, where we summarize the significant contributions of this research to the field of TTS. Our work not only advances the state-of-the-art in voice cloning and cross-lingual adaptation but also opens new avenues for personalized TTS applications, particularly in resource-constrained settings.