基於知識診斷之動態大型語言模型編輯框架

Abstract

從頭開始訓練大型語言模型（LLMs）是一項昂貴的工作，特別是在世界知識不斷演變的背景下。為了保持 LLMs 的相關性與準確性，模型編輯已成為一個關鍵的研究領域。然而，儘管這些方法充滿潛力，但其成效與副作用背後的潛在因素，特別是知識本身的特性如何影響編輯結果，仍然大多未被探討。本論文對此進行了系統性的研究，深入分析了知識的三個核心維度——流行度、模型對其的熟悉度 以及問題類型 ——如何影響模型編輯的成功率與穩定性。我們的實驗結果一致地表明，編輯「流行」或模型「未知」的知識，成功率顯著更高；而不同的問題類型，尤其是「Why」與「Which」之間，也表現出截然不同的編輯難度與副作用程度。此外，本論文揭示了一個在傳統問答評測中無法發現的、全新的副作用，我們將其命名為「生成性失語症」。此現象表現為，即使模型在事實層面已成功編輯，其長文本生成能力卻可能嚴重受損，出現內容不連貫或重複等問題。這一發現證明了現有評估方法的不足之處。基於上述洞見，本論文進一步提出並驗證了一個「知識診斷編輯框架」，該框架能根據知識的特性，智能地調整編輯策略，從而有效提升困難案例的編輯成功率。總體而言，本研究不僅為模型編輯的成敗提供了系統性的解釋，也為未來發展更可靠、更安全的編輯技術與評估標準，指出了明確的方向。

Training large language models (LLMs) from scratch is an expensive endeavor, particularly as world knowledge continually evolves. To maintain the relevance and accuracy of LLMs, model editing has emerged as a pivotal research area. While these methods hold promise, they can also produce unintended side effects, and the influence of the target knowledge's characteristics on editing outcomes remains underexplored.

This thesis delves into these critical factors by systematically analyzing how the nature of knowledge---specifically its popularity, the model's prior familiarity with it, and the question type used to elicit it---affects the success and stability of model editing. Using the RealTimeQA dataset with representative editing methods on LLaMA-3 models, our experiments reveal several key findings. We demonstrate that edits involving popular or previously unknown knowledge are significantly more successful, and that question type has a profound impact on both editing efficacy and the severity of side effects .

Furthermore, while state-of-the-art methods like AlphaEdit exhibit perfect stability on standard question-answering benchmarks, we identify a novel and critical failure mode in more complex, long-form generative tasks, which we term "Generative Aphasia" . This phenomenon, characterized by disruptions in coherence and fluency despite accurate factual recall, reveals that conventional evaluation metrics are insufficient.

Based on our findings, we propose and validate a Knowledge-Diagnostic framework that adapts the editing strategy to the difficulty of the target knowledge, significantly improving performance in challenging cases. Our work underscores the necessity of a more nuanced, context-aware approach to model editing and calls for the adoption of more comprehensive evaluation protocols that include generative tasks to ensure truly robust and reliable knowledge integration.