StreaMeme：基於大型語言模型之直播迷因類別推薦

Abstract

迷因能夠傳達情緒與幽默感，並在直播過程中有效促進觀眾之間的互動及參與。然而，目前鮮少有研究協助直播主在直播環境中適時展示合適的迷因。由於需整合多模態線索、具備對直播情境的細緻理解，在直播過程中即時推薦相關迷因是一項極具挑戰性的任務。為此，本研究提出一套創新的系統 StreaMeme（liveStream Meme category recommender），旨在於直播過程中為直播主推薦合適的迷因類別。StreaMeme 利用視覺語言模型（Visual Language Model, VLM）以分析迷因的幽默感與情緒，並透過大型語言模型（Large Language Model, LLM）理解直播情境，進而推理出推薦合適迷因類別之理由。本系統所訓練之大型語言模型，可有效利用直播主之口述內容與觀眾留言，生成迷因解釋及直播情境推理，並將其輸出用於迷因類別推薦。實驗結果顯示，在真實直播資料中，StreaMeme 之 F0.5 分數優於多種基準模型，包含直接使用 LLM 提示及微調後的語言模型。

Memes convey sentiments and humor, and are useful to stimulate interactions and engagements of audiences during livestreaming. However, little research aids streamers in showing appropriate memes in livestreaming environments. Suggesting relevant memes timely in a livestream poses significant challenges as it requires multimodal cues and a nuanced understanding of the livestream context. This study proposes StreaMeme (liveStream Meme category recommender), a novel system that recommends meme categories for streamers during livestreaming. StreaMeme employs a visual language model (VLM) to analyze memes’ humor and sentiments, and digests livestreaming contexts with a large language model (LLM) to reason for meme recommendations. An LLM is fine-tuned to exploit streamer speeches, audience messages, meme explanations, and recommendation reasons; its output is then used for our meme category recommendation. Experimental results on real-world livestreams show that StreaMeme outperforms several baseline methods, including direct LLM prompting and fine-tuned language models in terms of the F0.5 scores.