結合 3D 高斯潑灑與大型語言模型於文物之對話展示

Abstract

本文提出一種創新的文物展示方式，設計了一個能讓文物「說話」的虛擬實境系統。不同於傳統由虛擬導覽員解說的方式，本系統賦予文物聲音與個性，讓文物具備了對話的能力，而不只是靜態展示品。我們首先利用 3D 高斯潑灑技術重建真實且高品質的文物模型，接著結合本地部署的大型語言模型讓文物能夠根據輸入的語音產生回應。我們亦導入檢索增強生成技術，讓模型能引用正確資料以提升回答的正確性。為了找出最適合展現文物個性以及產生生動回應的大型語言模型，我們對不同的繁體中文大型語言模型進行了比較，也分析了檢索增強生成對於提升回答正確性的效果。最後，我們於頭戴式裝置上測試 3D 高斯潑灑文物模型的視覺表現，探討高斯點數量、文物大小、與觀看距離對渲染順暢度的影響。

We present a new approach to the exhibition of cultural artifacts by designing a system that enables the artifacts to speak in virtual reality. Unlike traditional virtual museum guides, this system gives artifacts their own voices and personalities, allowing them to engage in conversation instead of just being silent objects on display. We first reconstruct high-quality artifact models with 3D Gaussian Splatting. Then we integrate a locally deployed large language model to generate responses from speech input. We also incorporate Retrieval-Augmented Generation (RAG) to improve the correctness of the responses by allowing the model to reference relevant context. We compare different Traditional Chinese large language models to identify the best for generating vivid and characterful responses, and we analyze the effectiveness of RAG in enhancing response quality. Finally, we evaluate the visual performance of artifact models on a VR headset, examining how splat count, artifact size, and viewing distance affect rendering performance.