從日本漫畫「看見」感官：漫畫如何使用圖象、符號及視覺隱喻再現感官經驗

Abstract

近年來許多研究者常將視覺比擬成一種可以用來溝通及反應對於世界認知的「語言」(Cohn, 2020; Forceville, 2005, 2012; Hsu & Chiang, 2021; Kövecses, 2019; McCloud, 1993; Schilperoord & Maes, 2009; Teng & Sun, 2001; Wallestad, 2013)。有感於此，本論文將研究漫畫（manga）如何使用視覺元素及各種表現形式來傳達跨感官──聽覺、觸覺、嗅/味覺。本論文試圖解釋下列三個研究問題：(1) 在表現感官經驗時，漫畫中的文字及文字是如何互動的？ (2) 漫畫是如何以視覺來呈現感官經驗？以及 (3) 這些視覺元素呈現各感官經驗時，背後是否有相關的認知及心理學之佐證？ 本研究提出一個新的理論架構──「視覺元素及表現形式模型（Visual Elements and Manifestations Model, VEMM）」以解決上述的研究問題。這個模型可以以更為宏觀的角度來分析漫畫中各格小至視覺元素（圖象、符號、視覺隱喻），大至如場景、觀點、文字、畫格、間白、敘事及排版的各種視覺表現形式。 根據「視覺元素及表現形式模型」，本論文提出三個重要的發現：首先，本研究對於認知語言學提出了新的貢獻──藉由比較文字與圖象在漫畫之間的互動，得以知道文字及圖象在表現感官經驗上的異同。研究指出文字和圖象在下列三個面向上有明顯的差異：表現情感與內在經驗的深淺、表現背景知識及場景的能力，以及隱喻及借喻的功能。 再者，本研究也發現「視覺元素及表現形式模型」中的視覺元素及表現形式在表現感官經驗時各有其特殊性。圖象是漫畫最核心且基礎的視覺元素，用以建立背景知識及脈絡。當圖象被作為借喻使用時，它可以指示出該感官經驗的類別性感受。符號，在漫畫中又稱為「漫符」，常被用來提醒讀者漫畫該格存在著「感官」經驗。視覺隱喻雖然出現的次數最少，但它卻蘊含著豐富的情感或評價的「價（valence）」。在感官經驗使用視覺隱喻時，可以增加這個經驗的強度。而這三個視覺元素分別在視覺敘事中扮演不同的角色──當這些視覺元素被作為借喻時，它們通常在「建立階段（establisher）」出現以用來建立背景知識；而視覺隱喻通常會扮隨著高強度的情緒影響力，並運用於「高潮階段（peak）」。 最後，研究藉由分析除視覺外四種感官經驗的視覺元素及表現形式，補足了過去感官表現在視覺呈現上的不足。論文也試圖以認知或心理學的角度來分析這四種感官經驗的視覺元素及表現形式異同。結果顯示，我們因為對於這些感官的熟悉度及認知不同，所以對於比較熟悉的感官經驗可能會用更為直接的視覺元素（如圖象）而少用視覺隱喻。反之，對於嗅/味覺這類的感官經驗，可能更常使用視覺隱喻來表現。同時，在陳述每種感官經驗時使用的敘事手法也有差異。例如，觸覺相關的敘事通常比較短小；但嗅/味覺的敘事通常篇幅較完整也較長，甚至可以呈現一段完整地品嘗美食或美酒的經驗。 本研究是第一篇使用漫畫來研究我們對於感官經驗的認識，期待這份跨學科的研究能夠貢獻於認知語言學、視覺，以及感官的相關領域。

Visual is a powerful language to communicate with others and reflect our conceptualization of the world (Cohn, 2020; Forceville, 2005, 2012; Hsu & Chiang, 2021; Kövecses, 2019; McCloud, 1993; Schilperoord & Maes, 2009; Teng & Sun, 2001; Wallestad, 2013). This thesis investigates how manga uses visual elements and manifestations to convey a cross-sensory experience, including hearing, touch, taste, and smell. There are three research questions to solve in this thesis: (1) What are the interactions between visuals and verbal language when presenting sensory experiences in the manga? (2) How do visuals in manga represent sensory experience? Moreover, (3) What are the cognitive/psychological explanations of each visualized sensory experience in the manga? In order to solve these research questions, this thesis proposes a new theoretical framework, the Visual Elements and Manifestations Model (VEMM), to provide a holistic view of analyzing the visual elements and manifestations in the manga. The VEMM can identify from the minor visual components, pictures, non-pictorial icons, and metaphors to the bigger and more complex manifestations such as scene, perspective, word, panel, gutter, narrative, and layout. By applying the VEMM, the thesis can deliver three main findings. Firstly, based on the interactions of the words and visuals, the thesis identifies how the verbal and the visual represent sensory experiences. Comparing the similarities and differences between verbal and visual provides more insights into the existing linguistics research. The result shows three main differences between visual and verbal: the perspective of presenting emotions and inner stages, the capability to show the background knowledge and scenes, and functions and sources of verbal and visual metaphors and metonymies. Secondly, the thesis uses VEMM and focuses on each visual element’s function and occurrence while describing the sensory experience. Apart from word usage, each visual element has its functions and tendency. Pictures are the fundamental elements in manga and are used to create background knowledge and contextual information. A picture with metonymical usage is used as an indicator to give readers categorical information of senses. The non-pictorial icons, or so-called manpu, are used to remind readers of the existence of sense in the manga. Metaphor, although it has the lowest occurrence, provides emotional and evaluative valence to the sensory experience. Adding metaphors can elevate the impact, and each visual element plays different role in the visual narrative. The metonymical pictures occur to provide background knowledge, and the metaphors accompanying peak show emotional solid impacts. Finally, the research bridges the gap by comparing the four senses’ visual elements and manifestations via VEMM. By comparing the usage of visual elements in each sense, it is also to investigate the cognitive and psychological reasons behind the usage. The result shows that we have different conceptualizations and familiarities with each sense. For instance, people seem more familiar with touch, so there is a more straightforward usage of pictures but less usage of metaphors. On the other hand, people use many metaphors, sometimes cross-modality metaphors, to describe the taste/smell-related experience. Furthermore, it seems that each sense’s visual narratives are different. For example, the visual narrative related to touch is shorter. However, the visual narrative is more extended when describing taste or smell-related experiences. Sometimes the long narrative also corresponds with the complete procedure of cuisine-tasting or wine-tasting. As the first research uses manga as research material to explore our overall conceptualization of sensory experience, the thesis expects to shed light on the interdisciplinary studies on visual languages and senses and contribute to our overall understanding of different senses.