應用人工智慧與聊天機器人於聾啞人的互動系統開發

Abstract

聽障與語言障礙者在日常溝通中持續面臨顯著挑戰，同時現代社會對個體情感需求的滿足亦日益受到重視。為應對此雙重需求，本研究旨在設計並實現一套創新的智慧互動框架。此框架以行動應用程式為核心交互介面，致力於改善聽障與語言障礙人士的溝通可及性與有效性，並為一般使用者提供富有同理心的情感支持及便捷的個性化生活輔助，從而全面提升用戶生活品質，積極促進社會包容性與個體福祉。 系統關鍵技術與功能實現如下：整合人臉辨識與情緒辨識技術，以增強人機互動感知與個性化體驗；開發多語言對話系統，透過融合GPT-4o模型及應用提示工程，實現中、英、台語的自然語言理解與生成。此外，本研究開發了一款界面簡潔直觀的跨平台行動應用程式，並針對聽障或無法言語者的特殊需求深度優化，用戶可藉此與具備個人助理及情感陪伴功能的聊天機器人互動。為進一步強化對聽障用戶的溝通輔助，系統集成了文字轉手語影片與語句潤飾功能，並初步具備處理多模態視覺與語言輸入的潛力。經由多樣化情境（包括一般用戶語音互動及聽障用戶文字互動）的驗證，初步結果顯示本系統在溝通輔助方面表現良好，且用戶接受度高。

Individuals with hearing and speech impairments face persistent significant challenges in daily communication, while the fulfillment of individual emotional needs in modern society is also gaining increased attention. Addressing these dual needs, this research designs and implements an innovative intelligent interaction framework. This framework, with a mobile application as its core interactive interface, aims to improve communication accessibility and effectiveness for individuals with hearing and speech impairments, while also providing general users with empathetic emotional support and convenient personalized life assistance, thereby comprehensively enhancing user quality of life and actively promoting social inclusivity and individual well-being. Key technological and functional implementations are as follows: integration of facial recognition and emotion recognition technologies to enhance human-computer interactional perception and personalization; development of a multilingual dialogue system, achieving natural language understanding and generation in Mandarin, English, and Taiwanese through the fusion of the GPT-4o model and application of prompt engineering. Furthermore, this study developed a cross-platform mobile application with a clean and intuitive interface, deeply optimized for the specific needs of users who are deaf or non-speaking. Users can interact via this application with a chatbot equipped with personal assistant and emotional companionship functions. To further strengthen communication support for users with hearing impairments, the system integrates text-to-sign-language video and text refinement features, and possesses nascent capabilities for processing multimodal visual and linguistic input. Validation through diverse scenarios (including voice interactions for general users and text-based interactions for users with hearing impairments) indicates that the system performs well in communication assistance and has high user acceptance.