應用連續情緒辨識於輕度認知功能障礙者的人機互動架構

Abstract

在本論文中，我們設計了一個可以為老年人或輕度認知障礙者提供情感支持（聊天、陪伴、分享情緒）的人機互動架構來減緩失智症的進展。在我們的人機互動架構之下，我們利用參與度模型來檢測人類的參與度。另一方面，我們結合臉部表情、語音和文字來檢測人類的情緒。針對臉部表情部分，我們提出了空間注意力機制沙漏卷積神經網絡 （SA-HCNN）模型，並結合 Transformer Encoder模型針對空間和時序特徵進行學習。針對語音部分，我們使用滑動窗口方法來提取梅爾頻率倒譜係數（MFCCs）特徵，並且建立一個 Transformer Encoder模型來提取時序特徵。針對文字部分，我們使用 Google Cloud Natural Language API獲得情緒分數。隨後，提出了一種機器人情緒生成系統，該系統使用人類情緒、人類參與度和機器人語音內容作為輸入，為機器人生成適當的情緒與行為。 此外，我們提出了基於人類情緒和參與度的HRI失智症狀指標，使機器人能夠在和人類互動的過程中檢測早期的失智症狀。我們同時將此HRI失智症狀指標與其他基於問答的認知任務相結合，以增強對失智症的評估。 我們提出的架構實現於本實驗室研發的服務機器人Mobi。我們招募了九位60歲以上的老年人與 Mobi進行互動。實驗結果表明，我們的人機互動架構使老年人和機器人互動時的體驗更加愉快且放鬆。

In this thesis, we designed a human–robot interaction structure that can provide emotional support (chatting, company, sharing mood) for older adults or people with mild cognitive impairment to prevent their dementia progression. Our engagement model detected human engagement levels, and human emotions were detected using a unique combination of facial expression, speech, and text. For facial expression modality, we proposed a spatial attention hourglass convolutional neural network (SA–HCNN) model and used the Transformer Encoder model to simultaneously compute the spatial and temporal information. For speech modality, we used the sliding window method to extract Mel-Frequency Cepstral Coefficients (MFCCs) features and built a Transformer Encoder model to extract temporal features. For text modality, we used the Google Cloud Natural Language API for sentiment analysis to obtain sentiment scores and magnitudes. Afterward, a robot emotion generation system that uses human emotions, human engagement levels, and robot speech content as inputs was proposed to generate appro-priate emotional states and expressions for the robot. In addition, we proposed HRI dementia symptom metrics based on human emotions and engagement levels to enable robots to detect early-stage dementia symptoms through interaction and combine them with several question–answer-based cognitive tasks to enhance the assessment of dementia. The proposed architecture was implemented via a service robot constructed by the NTU Robotics Laboratory called Mobi. We recruited nine older adults over the age of 60 to interact with Mobi. The experimental results showed that older adults had more enjoyable experience interacting with robots under our human–robot interaction structure.