基於凍結預訓練文字影像模型骨幹的一階段高效率開放詞彙語義分割

Abstract

開放詞彙語義分割因需在實際應用中識別訓練階段未見過的類別，近年來成為電腦視覺領域的重要研究課題。現有主流方法多採用兩階段架構：首先由遮罩產生器生成候選遮罩，再將這些遮罩輸入預訓練的對比影像文字模型進行視覺與語言的特徵匹配，以確定每個遮罩對應的語義類別。然而，此類方法在推理階段需對同一影像反覆抽取特徵，導致記憶體消耗較大且推理延遲明顯增加，限制其在實務環境中的應用效率。雖然現行一階段架構的模型其推理速度相較於二階段模型有顯著優勢，但部分方法在訓練過程中所需的時間與資源甚至高於二階段模型。 本研究旨在改善此問題，採用一種高效率的一階段開放詞彙語義分割方法，期望在加快訓練速度的同時，仍能維持穩定且良好的推理效能。將預訓練的對比影像文字模型直接作為模型的骨幹，並在訓練階段凍結其權重，僅更新遮罩產生器模組。此方法能夠減少訓練時間與顯著降低記憶體消耗。在推理階段，使用遮罩有效性篩選機制，透過池化處理各遮罩，以判斷其是否有效，從而避免無效遮罩影響最終的分割品質。實驗結果顯示，本研究提出的架構在精度與速度上皆可與現有主流方法媲美，並顯著改善推理效率與實用性。

Open-vocabulary semantic segmentation has become an important research topic in the field of computer vision due to the necessity of identifying unseen categories during training in real-world applications. Existing mainstream approaches typically adopt a two-stage architecture, which first generates candidate masks using a mask generator and then feeds these masks into a pre-trained contrastive vision-language model, e.g., CLIP, to perform visual-language feature matching, determining the semantic category for each mask. However, these methods repeatedly extract features from the same image during inference, resulting in significant memory consumption and latency, thereby limiting their efficiency in practical deployment. Although current one-stage models offer significantly faster inference compared to two-stage models, some methods require even more training time and computational resources than two-stage counterparts. This study aims to address the above issue by adopting an efficient one-stage openvocabulary semantic segmentation approach, with the goal of accelerating training while maintaining stable and reliable inference performance. A pretrained contrastive visionlanguage model is directly employed as the backbone of the architecture, with its weights frozen during training, updating only the mask generator module. This approach effectively reduces training time and significantly reduces memory usage. During inference, a mask validity filtering mechanism using mask pooling is implemented to determine the effectiveness of each predicted mask, thereby preventing invalid masks from negatively affecting segmentation quality. Experimental results demonstrate that the proposed architecture achieves performance comparable to those of existing mainstream methods in both accuracy and speed, while substantially improving inference efficiency and practicality.