邁向複雜3D視覺建構： 從物件層級補全與生成，到場景層級跨模態定位與修復

Abstract

近年來，3D電腦視覺的進展顯著提升了我們對複雜真實場景的理解能力。然而，真實世界的3D資料常因視角差異而變得不完整或存在跨視角的歧義，為穩健的3D建模帶來巨大挑戰。 本論文構建了一個統一的複雜3D視覺建模框架，研究分為物件層級與場景層級兩大任務。在物件層級中，首先解決了部分點雲的語義感知補全，以重建完整的幾何結構與語義標註；接著擴展單一模態生成至跨模態合成，將文字描述生成為3D模型。在場景層級中，我們示範了透過自然語言輸入精確定位3D場景中的目標物件，並在此基礎上執行場景級修補，移除指定物件後以多視角一致性技術填補缺失區域。這四大主題的系統整合顯著提升了建模的穩定性、生成品質、定位準確度與跨視角一致性，為沉浸式計算與機器人應用奠定了堅實基礎。

Recent advancements in 3D computer vision have significantly enhanced our capability to interpret complex real-world scenes. However, real-world 3D data often exhibit incompleteness and ambiguity across viewpoints, presenting considerable challenges for robust 3D modeling. This thesis focuses on constructing a unified framework for complex 3D visual modeling, organized into two main categories: Object-Level and Scene-Level. In the Object-Level sections, we first explore 3D Modeling via Semantic-Aware Completion, reconstructing 3D objects with complete geometry and semantic labels from partial point clouds and semantic information; then we examine 3D Modeling via Cross-Modal Generation, extending single-modal point cloud-to-point cloud transformations to text-driven 3D content synthesis. For Scene-Level sections, we start from introducing 3D Modeling via Scene-Level Visual Grounding, demonstrating how natural language prompts can precisely identify target objects within 3D scenes; finally, with the capability of locating any object in a given 3D scene, we present 3D Modeling via Scene-Level Inpainting, showcasing methods to remove specified objects and coherently fill missing regions using multi-view consistent techniques. Through the systematic integration of these four interconnected themes, our research demonstrates significant advancements in modeling robustness, generative fidelity, localization precision, and cross-view consistency, establishing a solid foundation for future applications in immersive computing, robotics, and beyond.