以三維門卷積與時序性補丁對抗式生成網路之任意形狀影片修復

Abstract

任意形狀的影片修復是一項很有挑戰性的任務，可以廣泛用於影片編輯（如字幕移除、物體移除）。現有基於補丁修補的方法無法處理非重複性結構（如人臉），而將圖像修復模型直接應用於影片則會導致影片前後不一致。本篇論文提出一種基於深度學習的任意形狀影片修復模型，以三維門控卷積來應對任意形狀遮罩的不確定性，並加上時序性補丁對抗式生成網路（Temporal PatchGAN）來增強影片的一致性。此外，我們收集影片並設計一種任意形狀遮罩生成的演算法，以建立任意形狀影片修復（FVI）資料集，用於訓練和評估影片修復模型。在FaceForensics和FVI資料集上進行的實驗結果顯示，我們的方法優於現有的方法。相關程式碼、結果影片和FVI資料集都在Github上開源 https://github.com/amjltc295/Free-Form-Video-Inpainting。

Free-form video inpainting is a very challenging task that could be widely used for video editing such as text removal. Existing patch-based methods could not handle non-repetitive structures such as faces, while directly applying image-based inpainting models to videos will result in temporal inconsistency (see videos http://bit.ly/2Fu1n6b). In this paper, we introduce a deep learning based free-form video inpainting model, with proposed 3D gated convolutions to tackle the uncertainty of free-form masks and a novel Temporal PatchGAN loss to enhance temporal consistency. In addition, we collect videos and design a free-form mask generation algorithm to build the free-form video inpainting (FVI) dataset for training and evaluation of video inpainting models. We demonstrate the benefits of these components and experiments on both the FaceForensics and our FVI dataset suggest that our method is superior to existing ones. Related source code, full-resolution result videos and the FVI dataset could be found on Github https://github.com/amjltc295/Free-Form-Video-Inpainting