台灣華語圓唇元音感知中的體感與視覺影響

Abstract

語音是多模態的，不僅概括視覺還有體感的資訊來幫助我們對於語音理解。唇部作為多模態的發音器官，提供了一個很好的機會來探索這些感官系統之間的複雜交互作用。本論文旨在研究多模態唇部資訊是否影響台灣華語中的高圓唇元音的感知辨識。此外，還探討在高圓唇元音情境下的跨模態資訊整合及交互作用。每項實驗招募了三十位台灣華語母語者，分別進行三個實驗以檢驗：1）體感回饋對聽覺感知的影響，2）視覺資訊對聽覺感知的影響，以及3）體感和視覺資訊的一致性對聽覺感知的影響。受試者透過進行內在語言表現作為體感回饋（實驗一）、觀看視覺元音影片（實驗二），或在觀看視覺元音影片的同時進行內在語言表現（實驗三），以辨識從 /u/ 到 /y / 連續體的聽覺刺激。結果顯示，儘管 /u/ 和 /y/ 在舌位上有本質的不同，體感回饋和來自唇部的視覺資訊都能顯著影響高圓唇元音 /u/ 和 /y/ 的辨別。當兩種模態提供一致的信息時，跨模態資訊的交互作用顯示出正向的累加效果。此外，當兩種模態的信息不一致時，體感對比較大的人可能會更依賴體感回饋，而體感對比較小的人可能會更依賴視覺信息。總體來說，這些發現有助於我們理解語音感知以及揭示多模態整合本身的複雜性。

Speech is multimodal, incorporating not only vision but also the somatosensory system to enhance understanding. The lips, being a multimodal articulator, offer a unique window to explore the complex interactions across these sensory systems. This thesis focuses on investigating whether multimodal lip information influences the identification of high rounded vowels in Taiwan Mandarin. Additionally, it aims to explore the interactions of cross-modal information under the context of high rounded vowels. Thirty native Taiwan Mandarin speakers were recruited for each of the three experiments to examine: 1) the effect of somatosensory feedback on auditory perception, 2) the effect of visual information on auditory perception, and 3) the congruent effect of somatosensory and visual information on auditory perception. They were instructed to perform inner speech as somatosensory feedback (Experiment 1), watch visually articulated vowel videos (Experiment 2), or perform inner speech with the presence of visually articulated vowel videos (Experiment 3) for the respective three experiments while identifying the auditory stimuli from /u/ - /y/ continua. The results showed that both somatosensory feedback and visual information from the lips can significantly influence the categorization of the high rounded vowels /u/ and /y/, despite their intrinsic differences in tongue position. The interaction of cross-modal information showed an additive effect when congruent information from the two modalities was provided. Additionally, individuals with greater somatosensory contrasts may rely more on somatosensory feedback when incongruent information from the two modalities was provided, whereas individuals with lesser somatosensory contrasts may rely more on visual information in the same conditions. Overall, these findings contribute to our understanding of how we perceive speech, shedding light on the complexity of multimodal integration.