手機影片蒐尋與辨識

Abstract

近幾年，由於行動裝置的爆炸性成長與方便性，使用者需求已經顯著地從個人電腦轉移到手持裝置上。許多使用者們現在隨身攜帶他們的行動裝置且當空閒總是瀏覽線上內容 - 其中大部分為多媒體內容。因由搜尋相似或辨識影片使用者興趣的影片的需求，我們提供了辦法讓使用者使用行動裝置上的照相機與麥克風擷取影片然後得到使用者所感興趣的相關訊息。 在第一部分，我們呈獻了行動裝置上大量影片搜尋系統，取名為``Me-link'，漸進式的結合輕量的聲音和視覺的特徵。使用我們的系統，使用者只需要將行動裝置的照相機指向他們所感興趣的影片。系統會擷取畫面和聲音，然後立即檢索出影片的相關訊息。當使用者拍攝更久，系統會漸進式的聚集時間上的線索然後回傳更精準的結果。我們亦考慮現實環境的干擾，當使用者不一定能取得清晰的視覺或音覺訊號。在合成視覺或音覺線索階段，我們的系統自動的偵測有效的訊號再回傳最終的結果。在伺服器端，使用者可以經由網頁上傳影片語相關訊息。此外我們還連結了訊號流，因此使用者可以利用我們的``Me-link'系統取得即時的電視節目。 在第二部分，我們專注於有潛力於許多生活上應用的影片辨識系統，而且提出新穎的系統能夠在手持裝置運算能力與網路頻寬限制下即時的辨識影片。我們提出漸進式與快速中止的影片辨識。當使用者記錄或觀看影片時，系統結合聲音與視覺的線索，然後執行漸進式的影片辨識。使用快速中止的方法，系統明智地只使用少量早期的畫面辨識影片，此方法大量的節省計算與網路資源。我們的系統相較於原始影片特徵只需傳輸 7KB 的影片資訊，而且每三秒影片錄放時回傳辨識結果提供立即的反饋的信息。

In recent years, due to the explosive growth of mobile devices and the convenience of portable device, the consumer requirement significantly shifts from desktop computers to portable devices. Many people now carry their mobile devices all the time and browse online content – mostly multimedia content, whenever they have spare time. With the demand of searching or recognizing similar videos the videos of interest on mobile, we present solutions that uses the camera and microphone to capture the video and then retrieve the information that user is interested in. In first part, we present a scalable mobile video search system, named ``Me-link,' based on progressive fusion of light-weight audio visual features. With our system, users only have to point the mobile camera to the video they are interested in. The system will capture the frames and sounds, then retrieve relevant information immediately. As the users hold the mobile longer, the system progressively aggregates the cues temporally and then returns more accurate results. We also consider the real world noisy environment, where users may not get clear visual or audio signals. In the aggregation step of audio and visual cues, our system automatically detects the available channel for the final rank. On the server side, users can upload the videos with information via website. Besides, we also link the streaming signals so that users can get the real time broadcasting with ``Me-link'. In second part, we focus on video recognition which has great potential for many applications in our everyday life and present a novel system enabling real-time video recognition under the constraints of mobile computing power and network bandwidth. We propose the concepts of progressive and early-stop video recognition. As the user records/watches the videos, the system aggregates audio visual cues and conducts video recognition in progressive manner. With the early-stop method, the system recognizes videos intelligently using only the early shots, which greatly saves computational and network resources. Our system only transmits 7K bytes video information which mush smaller than original video features, and returns the recognized results every 3 seconds of video playback that provides instant feedback.