大型語言模型在語音到文字摘要生成任務中的評估: 基於會議語料庫的研究

Abstract

隨著現代企業和組織日益依賴會議來進行溝通和決策，會議記錄和摘要的生成變得尤為重要。手動撰寫會議記錄和摘要過程繁瑣且容易出錯，並且品質往往依賴於紀錄者的能力，因此過去便有相關研究使用語言模型進行會議摘要。而近年來，大型語言模型崛起，如 GPT、Gemini 和 Llama 系列，目前已經有先關研究證明大型語言模型在自然語言處理相關任務如: 文本生成、翻譯、問答系統等有優異的表現。會議記錄文本通常較長且結構複雜，包含多個發言者的對話，涉及話題廣泛，導致會議摘要的生成與一般文章摘要有所不同。然而目前針對使用大型語言模型生成會議摘要的研究仍然相對較少，因此本研究旨在填補這一空缺。 本研究評估多種大型語言模型在生成會議摘要方面的效果。通過使用 AMI 會議語料庫，並結合不同的預處理方法（如 Google 語音識別和 Whisper Base 轉錄）及不同的Prompt設計，來比較這些模型的表現。研究結果顯示，GPT-4 在大多數情況下表現最佳，但在需要高度精確率的情境中，GPT-3.5 更具優勢。而Gemini 1.5 Pro 在召回率方面表現突出。本研究提供使用不同大型語言模型在實際生成會議摘要時的建議，可以依據不同需求選擇相應解決方案，期望這些發現能夠幫助企業和組織選擇適合的技術來提高會議記錄和摘要的效率與準確性。

Enterprises and organizations increasingly rely on meetings for communication and decision-making, the generation of meeting summaries has become particularly important. The manual process of writing meeting summaries is cumbersome and prone to errors, with the quality often depending on the recorder's ability. Consequently, there has been related research using language models for meeting summarization in the past. In recent years, the rise of large language models such as GPT, Gemini, and Llama series has demonstrated exceptional performance in natural language processing tasks like text generation, translation, and question-answering systems. Meeting transcripts are usually lengthy and complex, involving multiple speakers' dialogues and covering a wide range of topics, making meeting summarization different from general article summarization. However, there is relatively little research on using large language models for meeting summarization, and this study aims to fill this gap. This study evaluates the effectiveness of various large language models in generating meeting summaries. By using the AMI meeting corpus and combining different preprocessing methods (such as Google Speech Recognition and Whisper transcription) with different prompt designs, and compare the performance of these models. The research results show that GPT-4 performs best in most cases, but GPT-3.5 is more advantageous in situations requiring high precision. On the other hand, Gemini 1.5 Pro excels in recall rate. This study provides recommendations for using different large language models in practical meeting summarization scenarios, allowing for the selection of appropriate solutions based on specific needs. It is hoped that these findings can help enterprises and organizations choose suitable technologies to improve the efficiency and accuracy of meeting minutes and summaries.