大型語言模型於專家標記任務之局限性

Abstract

資料標記旨在對資料進行相關訊息的標記或標籤化，大量研究報告的結果顯示利用大型語言模型作為人類標記者的正向潛在能力。然而，現有研究主要聚焦在經典的自然語言處理任務上，尚未充分探索大型語言模型在需要專家知識領域中作為標記者的表現。在本論文中，我們系統性的評估其在金融、生物醫學及法律三個高度專業化領域的專家級標記者的表現，研究辦法包括單一語言模型以及多語言模型合作在內的綜合方法，以評估其性能和可靠性。實驗結果表明，儘管大型語言模型作為標記者表現出一定的前景，但其在不同領域和任務中的表現有顯著的差異。從成本效益的角度來看，我們的分析表明大型語言模型在使用 vanilla 或 CoT 方法時，能夠有效節約成本並保持中庸的表現。然而，儘管其擁有這些優勢，大型語言模型在高度專業化的任務中尚不能完全替代人類專家。據我們所知，本論文為首篇系統性地評估大型語言模型作為專家級標記者表現的研究，提供在專業領域中的實證結果和初步見解。

Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of work has reported positive results on leveraging large language models as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we present a systematic evaluation of LLMs as expert-level data annotators across three highly specialized domains: finance, biomedicine, and law. We investigate comprehensive approaches, including single LLMs and multi-agent LLM frameworks, to assess their performance and reliability. Our experimental results reveal that while LLMs show promise as cost-effective alternatives to human annotators, their performance varies significantly across different domains and tasks. From a cost-effectiveness perspective, our analysis indicates that LLMs, particularly when using the vanilla or CoT methods, offer substantial savings compared to traditional human annotation processes. Despite these advantages, LLMs are not yet a direct substitute for human experts in highly specialized tasks. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators, providing empirical results and pilot insights in specialized domains.