三維可供性之重建基於自我視角示範影片

Abstract

發展具備泛化能力的機器人技能至今仍極具挑戰性。受到心理學啟發，「可供性」（affordance）已被視為一種具潛力的中介表徵，可用來引導機器人進行物體操控。然而，多數現有研究主要聚焦於來自影片的二維可供性，忽略了攝影機位置、絕對空間座標、深度與幾何結構等關鍵空間資訊。為此，本研究提出一種無需訓練的創新方法，可從第一人稱操作示範影片中建構三維可供性。針對第一人稱影片中缺乏靜態高品質畫面而導致三維重建困難的問題，我們採用三維基礎模型 DUST3R，能在不使用 COLMAP 的情況下，從稀疏影像中重建場景。我們首先以手部偵測技術擷取接觸時間與二維接觸點，再透過 DUST3R 還原互動場景，並將接觸點以高斯熱圖投影至三維空間；同時，我們利用三維手部姿態估計取得手部軌跡，並透過線性回歸整合其時空動態，建構出完整的人物與物體互動歷程。實驗結果顯示，我們的方法能有效應用於 Ego4D-Exo 資料集中的七項真實世界料理任務，展現其於複雜操控場景中建構三維可供性的潛力。

Developing robots capable of generalized skills remains an exceedingly challenging task. Drawing from psychology, the concept of affordance has emerged as a promising intermediate representation to guide robot manipulation. However, prior work has primarily focused on 2D affordances from video, neglecting critical spatial information such as camera positioning, absolute position, depth and geometry. In this paper, we present a novel training-free method that constructs 3D affordances from egocentric demonstration videos. To address the challenge of insufficient static, high-quality frames for 3D reconstruction in egocentric videos, we employ the 3D foundational model DUST3R, which reconstructs scenes from sparse images without requiring COLMAP. We analyze videos using hand detection to identify contact times and 2D contact points, reconstruct these interactions using DUST3R, and project the 2D contact points into 3D space using gaussian heatmaps. Finally, we derive hand trajectories through 3D hand pose estimation and process them using linear regression to integrate the spatiotemporal dynamics of human-object interactions. We demonstrate the effectiveness of our method on the ego4d-exo dataset for seven real-world hand-object manipulation tasks in cooking scenes.