基於形狀感知的室內場景三維高斯散射重建與補全方法

Abstract

室內場景的三維重建涉及從捕獲的數據（如圖像或 LiDAR）創建物理空間的三維表示。此過程對於數位孿生創建、AR/VR 場景生成和室內導航等應用至關重要。雖然近期的神經表示方法如神經輻射場（NeRF）和三維高斯散射（3DGS）能夠實現逼真的新視角合成，而稀疏視角方法降低了視角要求，但現實世界的捕獲場景仍然受到遮擋和有限視角的影響，導致不完整的觀測，在重建場景中產生空洞和偽影。 本論文提出了一個三階段的室內三維場景重建與補全方法。首先，我們使用多線索監督進行初始場景重建，結合深度和法向量監督以確保精確的表面對齊並減少 3DGS 表示中的偽影。其次，我們通過分割和追踪將複雜場景分解為個別物體，使物體層級的補全變得可行。最後，我們進行物體層級的補全，使用幾何優先的方法將幾何重建與外觀合成分離。我們使用形狀補全模型從部分點雲重建缺失的幾何，然後引入形狀感知高斯遮罩自編碼器，修改遮罩策略，選擇性地對新補全的區域進行遮罩處理，同時將原始可見區域視為未遮罩輸入。解碼器隨後為這些新增區域重建完整的三維高斯屬性。 實驗結果表明，我們的方法優於現有方法。與目前專注於局部空洞填補或物體移除的 3DGS 修復技術，或僅完成網格表示而不生成完整 3DGS 輸出的混合方法不同，我們的方法產生完整的室內場景 3DGS 表示。

3D reconstruction of indoor scenes involves creating three-dimensional digital representations of physical spaces from captured data such as images or LiDAR. This process is crucial for applications in the digital twin creation, AR/VR scene generation, and indoor navigation. While recent neural representations like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) enable photorealistic novel view synthesis, and sparse-view methods like pixelSplat reduce viewpoint requirements, real-world capture scenarios still suffer from occlusions and limited viewpoints, resulting in incomplete observations that cause holes and artifacts in reconstructed scenes. This thesis presents a unified approach to reconstruction and completion for indoor 3D scenes through a novel three-stage pipeline. First, we perform initial scene reconstruction with multi-cue supervision, incorporating depth and normal supervision to ensure precise surface alignment and reduced artifacts in the 3DGS representation. Second, we decompose complex scenes into individual objects through segmentation and tracking, making completion tractable at the object level. Third, we employ object-level completion using a geometry-first approach that separates geometric reconstruction from appearance synthesis. We employ shape completion models to reconstruct missing geometry from partial point clouds, then introduce a Shape-Aware Gaussian-MAE that modifies the masking strategy to selectively mask only the newly completed regions while treating the originally visible areas as unmasked input. The decoder then reconstructs full 3D Gaussian attributes for these newly added regions. Experimental results demonstrate that our approach outperforms existing methods. Unlike current 3DGS inpainting techniques, which focus on localized hole filling or object removal, or hybrid methods that only complete mesh representations without generating full 3DGS outputs, our method produces complete 3DGS representations of indoor scenes.