增強 3D 場景理解的通用語意神經輻射場

Abstract

利用多視角輸入合成新視角圖像，神經輻射場（Neural Radiance Fields，簡稱NeRF）已成為三維視覺領域中的熱門研究課題。在本文中，我們引入了一種名為通用語義神經輻射場（Generalizable Semantic Neural Radiance Fields，簡稱GSNeRF）的方法，它在合成過程中獨特地考慮了圖像語義，從而可以為未見過的場景生成新視角圖像及其相關的語義地圖。我們的GSNeRF由兩個階段組成：語義幾何推理和深度引導的視覺渲染。前者能夠觀察多視角圖像輸入，從場景中提取語義和幾何特徵。後者在圖像幾何信息的指導下，執行圖像和語義渲染，具有更好的性能。我們的實驗不僅證實了GSNeRF在新視角圖像和語義分割合成方面優於先前的工作，而且進一步驗證了我們的採樣策略對視覺渲染的有效性。

Utilizing multi-view inputs to synthesize novel-view images, Neural Radiance Fields (NeRF) have emerged as a popular research topic in 3D vision. In this work, we introduce a Generalizable Semantic Neural Radiance Fields (GSNeRF), which uniquely takes image semantics into the synthesis process so that both novel view image and the associated semantic maps can be produced for unseen scenes. Our GSNeRF is composed of two stages: Semantic Geo-Reasoning and Depth-Guided Visual rendering. The former is able to observe multi-view image inputs to extract semantic and geometry features from a scene. Guided by the resulting image geometry information, the latter performs both image and semantic rendering with improved performances. Our experiments not only confirm that GSNeRF performs favorably against prior works on both novel-view image and semantic segmentation synthesis but the effectiveness of our sampling strategy for visual rendering is further verified.