互動式影片物件切割與影像搜尋

Abstract

隨著數位影片數量的成長，如何有效地獲取相關資訊成為今日相當重要的問題。消費者在觀看影片時，時常會對出現的物件有興趣，想進一步知道該物件的相關訊息。近年來觸控式面板的蓬勃發展，視覺化的操作方式大幅提升各類產品的互動性。我們提出一個互動式影片物件切割與搜尋系統，希望提供使用者感興趣物件的相關資訊。一般來說，影片的相關文字資訊，如標題、說明、字幕等等，不常與影片中出現的物件直接相關。我們以物件的視覺特徵搜尋外部影像資料庫，而影像伴隨的標籤足以提供這些影片物件更進一步的資訊。由於單一視角的物件常會受限於不同角度，而沒有充足的特徵去搜尋。因此，我們利用影片物件切割，改善搜尋的準確性。同時，為了滿足搜尋的及時性，我們使用影片壓縮的運動向量，改善物件切割的效率。實驗結果顯示，運動向量能大幅提升切割速率且維持搜尋的準確性。

The touch-based displays (devices) have entailed rich interactions between videos and users. The objects appearing in videos usually interest users in wanting to know relative knowledge about them. We propose a video playback system for users to interactively query objects of interest in videos. Since the text information accompanied with videos might not be strongly related to the object of interest, we adopt visual appearances as features to retrieve similar objects from large image collections. The tags associated with the retrieved images are used to reveal related information of the object of interest for further exploiting related knowledge. Solely relying on single viewpoint of the object to query may suffer from different poses, occlusions and is not robust. So we present a novel video object segmentation framework to improve retrieval precision, which is based on a graph cut segmentation. To ensure prompt response and effectiveness, we augment the graph cut algorithm with compressed-domain motion vectors; compared with the prior method, the processing speed of our segmentation framework is significantly improved. The experiments on community-contributed videos demonstrate the effectiveness of our segmentation framework based on multi-frame object region query and the improvement of retrieval precision.