基於腦波頻譜變化探討聆聽音樂之情緒反應

Abstract

腦波訊號是以非侵入量測的高時間解析度腦部訊息，其動態頻譜變化已大量的被用於探討與大腦功能相關之神經活動，並且可被用以特徵化或偵測大腦功能相關之腦波，以建構可應用於各種領域之腦機介面。現今，因為於多媒體欣賞的過程中，從腦波訊號中較能擷取與感知及情緒反應相關之訊息，因此以腦波誘發之多媒體研究為一逐漸熱門的主題，其主要的概念在於經由解析使用者腦波訊號中的多媒體誘發反應以評價其內容的重要性。為了達成此目的，如何特徵化與多媒體內容感知及情緒反應相關的腦波訊號即為一個非常關鍵的環節。然而，腦波頻譜動態變化與情緒反應兩者間之相關性及其用於情緒辨識之可行性尚未被廣泛的研究。 此論文主要專注於探究於聆聽音樂時情緒反應之腦波動態頻譜變化，為了更瞭解以腦波為基礎的情緒辨識領域，本論文的目的分為三個部分。首先，本論文主要建構一套最佳機器學習方法以腦波區分聆聽音樂時的不同情緒狀態。再者，本論文將採取獨立成份分析方法研究情緒反應，此方法近來於神經科學上被廣泛的用來分析腦部活性。最後，為了進一步提供音樂結構與情緒感知兩者之間的關聯性，本論文亦利用獨立成份分析方法研究與音樂調性及音樂速度感知相關的潛在神經機制。 本論文之發現將對於以腦波為基礎的情緒辨識領域產生許多有意義的貢獻。第一，本論文有系統的嘗試特定腦波特徵擷取及分類的方法以解決情緒分類的問題，結果顯示以非對稱多波段頻譜為特徵型式與以支援向量機為分類器的結合，於26位受測者中可達到最高的平均辨識率82.29% ± 3.06%，其四種情緒包括：歡樂、憤怒、悲哀及愉快。另外，一群由額葉及頂葉擷取的特徵被鑑定出最能提供與情緒處理相關且有區別度的資訊，其對於受測者變異不敏感，並大量的與先前文獻一致。第二，相較於前者以特徵為基礎的分類方法，本論文亦以獨立成份分析方法分離與音樂誘發情緒反應相關的獨立頻譜變化，結果發現等偶極子座落於額葉至中葉區域的獨立腦部活性所顯出的delta 及theta 頻譜變化是與受測者自評的情緒相關。此獨立的腦部活性較不受其他腦部活動的干擾，將能補充先前情緒感知相關的腦波研究。第三，本論文進一步利用獨立成份分析方法探究聆聽音樂時與音樂調性及音樂速度相關的獨立腦部訊號源，結果發現六個腦區顯出與音樂調性或音樂速度顯著相關的頻譜差異，包括其等偶極子座落於額葉內區、左側或右側的感覺運動區、頂葉上區、頂葉內區及兩側枕葉。這些區域與先前使用其他神經造影技術所得之音樂結構相關腦區一致。更重要的，其中額葉內區所顯出的delta 及theta 頻譜變化除了發現與音樂結構相關外，亦與本論文先前發現與情緒評價及情緒強度的證據相符。 總結，本論文除了依據機器學習方法成功的構建一套最佳的以腦波為基礎的情緒辨識機制外，亦依據神經計算方法探究與情緒反應及音樂結構(調性及速度)相關的腦波頻譜變化。本論文之發現將促進對於以腦波為基礎之情緒辨識研究的瞭解，改進及最佳化，並且取得更多的基礎以利於未來實現即時情緒辨識系統於多媒體應用中。

Ongoing electroencephalogram (EEG) provides noninvasive measurement of brain activity with temporal resolution in milliseconds. The spectral dynamics of EEG has largely been used to investigate the neural activity engaged in brain functions, and also to characterize/detect function-related EEG patterns to construct brain-computer interfaces in different fields. Nowadays, since EEG might provide more insights into the processes and responses of perception and emotional experience during multimedia appreciation, the EEG-inspired multimedia research has been a growing research topic. The main concept is to assess multimedia content from users’ brain signal through interpreting the induced content-related responses. Toward this end, how to characterize user’s perception and emotional experience in response to multimedia from EEG is very crucial. However, the correspondence of EEG spectral dynamics and emotional responses and its feasibility used for emotion recognition have not been extensively studied yet. This dissertation mainly focuses on exploring the EEG spectral dynamics of emotional responses in music appreciation. The overall objective of this dissertation is threefold for gaining more insight to the field of EEG-based emotion recognition. Firstly, this dissertation aims to construct an optimal machine learning approach to differentiate EEG patterns into distinct emotion states during music appreciation. Next, this dissertation is to adopt the independent component analysis (ICA), popularly used to analyze brain activity in neuroscience nowadays, to assess the emotional responses. Lastly, in order to provide new insights into the link between the changes in musical structures and the emotional perception, this dissertation is further to utilize ICA to investigate the underlying neural mechanisms which engage in musical mode and tempo perception. Several findings significantly contribute to the field of EEG-based emotion recognition. First of all, this dissertation has systematically conducted certain of EEG feature extraction and classification methods trying to solve the emotion classification problem. The results showed that combining a feature type of spectral power asymmetry across multiple frequency bands with a classifier of support vector machine (SVM) was an optimal way for characterizing four emotional states (joy, anger, sadness and pleasure) with an average subject-dependent accuracy of 82.29% ± 3.06% across 26 subjects. A group of features extracted from the frontal and parietal lobes have been identified to provide discriminative information associated with emotion processing, which were insensitive to subject-variability and largely consistent with previous literature. Next, in contrast to feature-based classification approach, ICA was used to separate independent spectral changes of the EEG in response to music-induced emotional processes. An independent brain process with equivalent dipole located in the fronto-central region exhibited distinct delta-band and theta-band power changes associated with self-reported emotional states, which were less interfered by the activities from other brain processes complement previous EEG studies of emotion perception to music. Lastly, by applying ICA to multi-channel scalp EEG data, this dissertation further explored temporally independent brain sources that contribute to the perception of musical mode and/or tempo during natural music listening. Six brain processes with equivalent dipoles located at or near the medial frontal, right/left sensorimotor, superior parietal, medial parietal, and lateral occipital areas exhibited statistically significant spectral differences in response to changes in musical tempo and/or mode. These areas were consistent with those previously reported brain regions, obtained by other neuroimaging modalities, associated with changes in musical structures. More significantly, the delta-band and theta-band activities projected from the medial frontal region were also found associated to emotional valence and arousal processes. In summary, this dissertation has successfully constructed an optimal EEG-based emotion recognition scheme based on feature-based classification approaches, but has evidently explored the EEG spectral dynamics associated to the emotional responses and musical structures (mode and tempo) based on neurocomputation approaches. All findings may facilitate the understating, improvement and optimization of the EEG-based emotion recognition research, but may get more fundamentals to implement a real-time emotion recognition system for multimedia applications in the near future.