基於多模態大型語言模型的情感圖片編輯

Abstract

Emotional Image Editing (EIE) 是通過圖片編輯來使圖片產生所需的情緒。為了便於使用者操作，使用者只需提供一張圖片和所需的情緒。這是一個尚未有太多研究的領域，現有方法受限於缺乏優質的資料集，以及無法參考使用者提供的圖片來決定編輯位置。本篇論文提出了一個基於Transformer和多模態大型語言模型（MLLM）的最先進的視覺情感分析（VEA）模型，用來建立一個包含能改變圖片情緒的指令的資料集。我們設計了EEdit，一個能進行情感圖片編輯的雙階段模型，由指令生成和圖片編輯兩部分組成。我們的模型在使用者心理學實驗中取得了最先進的結果。

Emotional Image Editing (EIE) involves modifying an image to evoke a desired emotion after editing. To facilitate user interaction, users only need to provide an image and the desired emotion. This task is relatively novel and underexplored, with existing approaches limited by inadequate datasets and the inability to reference user-provided images for determining edit locations. In this paper, we propose a state-of-the-art Visual Emotion Analysis (VEA) model based on transformer and Multimodal Large Language Model (MLLM) architectures, specifically the Multi-Branch Emotional Analysis Transformer (MEAT) and Emotion Question and Answer(EmotionQA), to create a dataset containing instructions capable of altering image emotions. The transformer-based model serves as the state-of-the-art VEA model, while the MLLM-based model aids in emotional image editing. We also introduce EEdit, a two-stage model for emotional image editing, comprising an instruction generation model and an image editing model. Our proposed model achieves state-of-the-art results in user psychological experiments.