基於Transformer之跨視角人體姿勢表徵學習及其於太極XR練習之應用

Abstract

人體姿勢理解是許多動作分析與互動應用中的關鍵技術。近期研究開始由2D骨架出發，學習具備3D語意且對視角變化具有不變性的嵌入表示(embedding)，以因應傳統3D重建方法在缺乏相機參數下產生的尺度與旋轉誤差，提升多視角場景中動作檢索、對齊與辨識的穩定性與泛化能力。

我們整合現有的Transformer架構與跨視角訓練策略，建構一個能同時捕捉人體動作之時序與空間特徵的學習框架。我們採用DSTformer作為骨幹模型，利用其空間與時間建模能力，從2D骨架序列中擷取關節間的空間關係與關節隨時間變化的時序特徵。模型透過三元組式(Triplet)的跨視角訓練機制，學習在不同視角下皆具一致性的姿勢嵌入表示。

為驗證方法效能，我們於跨視角姿勢序列檢索與影片對齊任務進行下游實驗，並進一步將所提模型應用於太極拳XR練習，讓使用者在不受攝影機視角限制的情境下進行學習，並在虛擬教練引導下即時獲得動作完成判斷之回饋，展現本方法於跨視角人體動作分析領域的實用性與應用潛力。

Human pose understanding is essential to many motion analysis and interactive applications. Recent studies have explored learning view-invariant representations from 2D skeletons, which capture 3D semantic information without relying on camera parameters. This direction aims to overcome the scale and rotation inconsistencies often encountered in traditional 3D reconstruction approaches, thereby improving the stability and generalizability of pose retrieval, alignment, and recognition tasks under multi-view scenarios.

In this work, we integrate an existing Transformer-based architecture with a cross-view training strategy to construct a learning framework that captures both temporal and spatial features of human motion. We adopt DSTformer as the backbone model due to its spatio-temporal modeling capabilities, enabling the extraction of spatial relationships between joints and temporal dynamics of joints from 2D pose sequences. The model is trained using a triplet-based cross-view learning scheme to ensure cross-view and semantically consistent embeddings.

To evaluate the effectiveness of our method, we conduct experiments on two downstream tasks: cross-view pose sequence retrieval and video alignment. Furthermore, we deploy our proposed model in Tai Chi XR practice, where users can practice under unconstrained camera viewpoints and receive real-time feedback on motion completion while following a virtual instructor. These results confirm the applicability of the proposed approach in realistic cross-view motion scenarios.