基於可變形對齊和光流傳播之實時注視點影片超解析度系統

Abstract

影片超解析度（VSR）技術能提升影片的解析度，為使用者提供更清晰的視覺體驗。然而，VSR 通常會因為較長的延遲，造成難以實時運行。此限制讓即時影片增強難以達成。 注視點串流利用了人類視覺系統（HVS）的特性，視覺敏銳度在注視點周圍較高，因此在注視點區域傳輸高解析度的內容；外圍區域對使用者的視覺影響較小，則使用較低解析度。此技術可以有效減少串流的傳輸頻寬和延遲並維持相同的視覺體驗。 在本論文中，我們提出了一個結合注視點串流概念的影片超解析度系統。透過眼動追蹤器獲取使用者的注視點，我們能將計算資源集中在注視區域，以實現實時的影片超解析度。我們也提出了一個可變形對齊與光流傳播模組，以有效對齊相鄰幀之間的特徵。此外，我們使用平移對齊來解決由於注視窗口位移所導致的錯位問題。我們的注視點影片超解析度系統大幅降低了超高畫質（UHD）影片超解析度的計算量，並將延遲減少約 90%，更保證了同等甚至更好的視覺體驗。此技術未來有望應用於配備眼動追蹤器的顯示設備中，為使用者提供超低延遲的解析度增強功能。

Video super-resolution (VSR) technology enhances the resolution of videos, providing users with a more precise visual experience. However, VSR often suffers from long latency and the inability to be performed online, making real-time video enhancement impractical. Foveated streaming leverages the human vision system's (HVS) characteristics by streaming high-resolution content around the viewport while using a lower resolution for peripheral areas, contributing less to the user's visual perception. This approach can be applied to reduce the transmission bandwidth and latency of streaming. In this thesis, we introduce a foveated video super-resolution system utilizing the concept of foveated streaming. Using an eye tracker to obtain the user's gaze position, we allocate computational resources to the focal area to achieve real-time online video super-resolution. We also propose a deformable alignment and flow propagation module to align features between adjacent frames efficiently. Additionally, shift alignment is employed to address misalignment caused by the movement of the foveated window. Our foveated video super-resolution system significantly reduces the computational load of ultra-high definition (UHD) video super-resolution, saving approximately 90% latency while ensuring the same or even better visual experience for the user. This technology has the potential to be applied in display devices equipped with eye trackers, providing users with ultra-low latency resolution enhancement.