人物交互檢測模型基於視覺語言模型轉移

Abstract

人物與物件交互（HOI）檢測是計算機視覺中的一項基本任務，其目標是在視覺場景中定位人物與物件對並識別它們的交互作用。本研究提出了一種新穎的HOI檢測框架，通過將視覺語言模型的先前知識轉移，實現更好的推廣能力。我們的框架包括一個混合式的結構，主要由兩個分支組成：第一個分支使用對比損失來改進從CLIP中提取的人物與物件之代表特徵向量，而第二個分支是一個分類器，使用交叉熵損失根據學習到的特徵向量進行預測。雙分支框架改善了人物與物件交互表示的品質和分類性能。為了提升HOI特徵向量的品質，我們提出了一個加強過的交互解碼器(Interaction Decoder)，其中包括額外的交叉注意力層(Cross Attention)。我們提出的這個新的解碼器能夠讓模型更容易從CLIP的視覺特徵圖中抽取帶有豐富資訊、有意義的區塊。通過將此解碼器與檢測主幹結合，使用知識集成block，我們實現了更精確的人物與物件對定位。我們的框架在完全監督且僅新增少量的模型參數下展現了優異的性能。

Human-object interaction (HOI) detection is a fundamental task in computer vision that objective to localize human-object pairs and identify their interactions in visual scenes. This work proposes a novel HOI detection framework that transfers prior knowledge from Visual Language Models to achieve better generalization. Our framework includes a hybrid network structure consisting of two branches: the first branch uses contrastive loss to improve the representation of human-object interactions extracted from CLIP, while the second branch is a classifier that uses cross-entropy loss to make predictions based on the learned features. The dual branch framework improves both the quality of representation of human-object interactions and the classification performance. In order to enhance the quality of interaction representations, we present an enhanced interaction decoder that incorporates an additional cross-attention layer. This new layer enables the extraction of informative localized regions from the visual feature map of CLIP. By integrating this decoder with the detection backbone using a knowledge integration block, we achieve more precise localization of human-object pairs. Our framework demonstrates superior performance in both fully-supervised scenarios and scenarios with limited parameters and data.