透過多階段的資料增強與回饋機制來優化大型語言模型的推理與可用性

Abstract

大型語言模型在各種任務中展現出卓越的能力，但其效能與可用性仍可透過精細優化進一步提升。本論文專注於大型語言模型訓練流程中的兩個關鍵階段：預訓練與後訓練。在預訓練階段，我們提出特定領域的資料整理與模型開發策略，以打造適用於特定語言與情境的大型語言模型，例如針對繁體中文的資料處理與評測方法及 Taiwan-LLM 模型。在後訓練階段，我們透過合成資料生成、偏好最佳化與迭代回饋來提升模型的能力，包括用於意圖識別的資料增強、用於自動評測與對齊的 LLM-Eval，以及用於逐步偏好最佳化的 Step-KTO。實驗結果顯示，合成資料增強可提升大型語言模型在低資源環境下的穩健性，而迭代回饋與精心設計的獎勵信號則能改善推理能力與輸出品質。本研究展示了一套完整的多階段強化框架，從預訓練到後訓練，以打造更符合語言、文化及實際應用需求的 LLM。

Large Language Models (LLMs) have demonstrated remarkable capabilities, but their performance, alignment, and trustworthiness can still be significantly improved through fine-grained optimization across multiple training stages. This thesis focuses on enhancing LLMs in two key stages: pre-training and post-training. In pre-training, we introduce domain-specific data curation and model development strategies to produce LLMs tailored for specific languages and contexts, such as Traditional Mandarin processing and benchmarking (e.g., the Taiwanese Mandarin Language Understanding benchmark and Taiwan-LLM). In post-training, we refine models using synthetic data generation, preference optimization, and iterative feedback, including methods like In-Context Data Augmentation for intent detection, LLM-Eval for automatic evaluation, and Step-KTO for stepwise preference optimization. Experimental results show that synthetic data augmentation enhances robustness in low-resource settings, while iterative feedback and well-designed reward signals improve reasoning and output quality. This framework demonstrates how multi-stage enhancements can create LLMs that are linguistically, culturally, and pragmatically aligned with user needs.