中文資訊檢索測試集設計與製作之研究

Abstract

摘要 在國內資訊檢索研究已日趨受到重視，合適的測試評估機制卻十分缺乏的背景下，本研究實際進行測試集的規劃與建置工作。首先透過相關文獻之閱讀與分析，觀察測試集的組成要素，並參考國外各測試集的結構、特性與建構經驗，訂定發展中文資訊檢索測試集的方法與程式。測試集建構工作主要包括蒐集整理檔、建立查詢主題、以及進行相關判斷三個部分。 本研究所建立的文件集來源為新聞網站中的五種電子報，共有132,207篇檔。查詢主題是透過網路問卷實際徵集查詢需求，並進行三次的篩選之後，修正建構而成，共完成50個查詢主題。相關判斷的部分則是先對每個查詢主題建立－相關文件候選集，再針對候選集中的每篇檔以人工進行相關判斷，每一查詢主題由三位次判斷者同時進行，最後，則依據判斷結果計算並定義文件的相關程度。 經由研究結果的分析顯示，就統計抽樣的觀點，本研究建構的檔集已具備一定的效度；查詢主題呈現詳盡且多樣化的查詢需求，能反映一些真實的檢索情況；三位判斷者的相關判斷結果則有顯著的一致性，推斷它們是具有可信度的。本測試集雖然尚處於初始階段，但已有完整的架構及一定的規模，未來的研究應可以此為基礎，作進一步的擴展與改進。

ABSTRACT The research and development of information retrieval (IR) has made much progress recently. However, there's not any applicable mechanism for system evaluation in the Chinese research society. This thesis aims at the design and the implementation for Chinese information retrieval benchmark. First, we observe the framework and contents of existing foreign benchmarks, and develop a realistic methodology, and setup a procedure to establish a Chinese IR Benchmark. Generally speaking, a benchmark consists of a set of documents, a set of topics, and a set of relevance between documents and topics. Accordingly, our task is also separated into three parts. The document set is downloaded from various electronic news sites, and totally 132,207 are collected. To build the topics, we investigate the real user information needs by using a questionnaire, then modify them to be the formal topics. As to relevance judgment, we first set up a pool of candidate documents for each topic, then invite three persons to judge the relevance. Finally, we combine the judgments and offer a relevance measure for each document in the pool. The result of our research shows that the quantity of the document set is valid from the viewpoint of sampling statistics. The topics reveal various information needs from users' viewpoint, and they may somehow reflect certain real situation while users proceeding their searches. Besides, the judgments of three persons have exhibited significant agreements, so we can say that the relevance judgment in our benchmark is reliable. Although the benchmark is in its first edition, it possesses a complete structure and medium scale, and we may further expand and improve it based on existing framework in the future.