ChatNTU: 檢索增強生成於校園場域的反問機制與資料檢索研究

Abstract

隨著大型語言模型的能力愈發強健，多輪對話式規章問答任務的主軸也從如何理解文句間的蘊含關係(entailment)、如何訓練模型產生自然語言的回覆或反問，轉變為如何使用大型語言模型提供更精確的答案以及更細緻、更多元的對話歷程。

本文以國立臺灣大學的校園問題為發想，實作了一個專注於校園場域常見問題的問答系統。我們探究了如何借助大型語言模型生成 HyDE、RHyDE 等 expansion 資料增加資料檢索的準確度，也設計了一個供問答系統反問使用者、適時擷取資料的框架，同時實驗了雜訊文章對於問答系統的效用。另外，因為實驗主題是臺灣大學，我們也收集了現實生活中師生會遇到的議題，並結合數百篇學校各處室的條文，彙集成一個力求忠實於現實情境的資料集。

總的來說，本篇論文的主要貢獻有二。一者，為製作了以校園場域為主題的多輪對話式問答資料集。二者，為設計了一個準確率提升、對話更細緻的問答系統框架。

As the capabilities of large language models (LLMs) continue to grow, the focus of multi-turn conversational rule-based question answering (QA) tasks has shifted from understanding entailment relationships between sentences and training models to generate natural language responses or follow-up questions, to utilizing LLMs to provide more accurate answers and more nuanced and diverse dialogue.

This paper presents a question-answering system tailored to address common inquiries within the National Taiwan University (NTU) campus. We explore the effectiveness of utilizing LLMs to generate expansion data, such as HyDE and RHyDE, to improve data retrieval accuracy. Additionally, we propose a framework that enables the question-answering system to engage in follow-up questions with users and dynamically retrieve relevant information. We also investigate the impact of noise passages on the system's performance. Since our work focuses on NTU, we carefully curated a dataset encompassing real-world issues encountered by students and faculty, along with hundreds of documents and regulations from various university departments, ensuring adherence to realistic scenarios.

Overall, this paper makes two main contributions. First, it presents a multi-turn dialogue question answering dataset focusing on the school domain. Second, it proposes a question-answering system framework that improves accuracy and generates more detailed dialogues.