通過結合建模和學習來恢復和增強圖像和視頻

Abstract

我們生活在一個豐富多彩、充滿活力和連續的世界中，人類視覺系統可以捕捉光線並將其傳遞給大腦進行進一步分析。十多年來，人們一直試圖使用數位相機來模仿人類的視覺系統並捕捉我們所看到的場景。然而，因為數位相機的不完美，充分獲取信息並反映人眼的感知具有挑戰性。因此，本論文的中心主題是彌合不完善的相機系統與人類視覺感知之間的差距。本論文的重點是圖像恢復和視頻增強，其中不完美的拍攝過程會破壞輸入圖像/視頻。我們嘗試使用基於深度學習的方法來解決以下任務：（1）單圖像 HDR 重建，（2）視頻幀插值，（3）多幀障礙消除，（4）視頻穩定，以及（5）空間-時間視圖合成。與將網絡視為黑盒並學習輸入-輸出映射函數的標準基於學習的方法不同，我們將物理約束或先驗項作為正則化來提高輸出質量和泛化能力。

We live in a colorful, dynamic, and continuous world. The human visual system can capture light and pass it to the brain for further analysis. People have tried to use the digital camera to mimic the human visual system and capture the scenes we saw for over a decade. However, with the imperfection of the digital camera, it is challenging to fully acquire the information and reflect the perception of the human eyes. Therefore, this dissertation's central theme is to bridge the gap between imperfect camera systems and human visual perception. This dissertation focuses on image restoration and video enhancement, where the imperfect capturing process corrupts input images/videos. We try to tackle the following tasks with learning-based approaches: (1) single-image HDR reconstruction, (2) video frame interpolation, (3) multi-frame obstruction removal, (4) video stabilization, and (5) space-time view synthesis. Unlike the standard learning-based approaches that treat the network as a black box and learn an input-output mapping function, we integrate physical constraints or prior terms as regularization to improve the output quality and the generalization ability.