應用多相機2D人體關節點偵測器於3D人體關節點空間之重建

Abstract

為了提升運動員在國際比賽上的表現，分析運動員在真實比賽中的狀態是必要的。因為含有標記的追蹤裝置會影響選手表現，本論文提供一個無標記的模型來重建運動員的3D人體姿態。模型中使用不同相機影像的2D人體姿態來進行三維重建。根據核面幾何(epipolar geometry)理論，兩台相機足以將影像座標投影到世界座標。如果使用多相機系統(>2)，可以利用更多雙相機組合配對來重建空間座標，提供更準確的世界座標。以克服可能來自於關節遮蔽或辨識錯誤所產生座標點的錯誤投影。因此，相機在空間中的位置和數量會決定場景的準確度。本論文討論相機在球場中的架設角度與關節點偵測率之間的關係。實驗上，以現有的OpenCV函式庫來建構多相機之間的幾何關係。本論文的2D人體姿態由深度學習模型(AlphaPose)來偵測。為了解決不同相機視角下的2D人體姿態準確度不一致的問題，結合核面幾何以及反投影誤差來得到多相機中的最佳的影像座標。最後以訊號後處理演算法來修正每個關節點於空間中隨時間的訊號。為了驗證我們的模型，本論文使用分別安裝在針對國立台灣體育運動大學(NTUS)球場上的三台高速相機來辨識投手姿勢。首先進行相機校準，我們使用二維的單平面棋盤格來計算相機的內部參數，並且利用三維的立方體來計算相機之間的相對參數。k-means聚類演算法被用來自動選取影像中適當的平面棋盤格，這可以防止內部參數在特定的影像區域下的過度擬和。使用K-fold來交叉驗證多相機系統的相對參數。三維訊號中，由於許多無法避免的辨識錯誤，挖除錯誤的特徵點可以使該訊號更接近真實訊號。本論文採用2D人體姿態的信心分數(cs)以及異常偵測(nd)來判斷錯誤的特徵點。訊號後處理上，使用傅立葉高頻濾波器以及高階多項式在時間序列上對3D數據擬合，可以減少訊號擾動以及修正不合理的點。”human_B20”右投的測試集中，對於右手腕、右手肘、右肩膀、左腰、左膝蓋以及左腳踝的關節點中，自動化的處理模型在平均每個關節位置誤差(MPJPE)達到15毫米以內。

To improve the performance of baseball pitcher in international competitions, it is necessary to analyze the pose of athletes in competitions. Because the player's performance will be affected by the tracking device with markers, this paper provides a markerless model to reconstruct the athlete's joint position in three dimensional space (3D). In this model, the two dimensional (2D) human pose from different camera images in different angle are used for the 3D reconstruction. The epipolar geometry theorem is used to project the object's global coordinate with at least 2 more cameras. In a multi-camera system (number >2), there would be more choices to get at least 2 joint points to reconstruct 3D coordinate, which could provide more accurate results. As we know, the joint occlusion point or the miss labeled joint point on the image will cause the wrong projection of the world coordinates. This paper first discussed the relationship between camera view angles and detection rate of joints. Based on the OpenCV function library and the deep learning model of 2D human pose estimation, and combined with the stereo vision theory, the geometric relationship between multiple cameras can be constructed. The deep learning model of AlphaPose is used for joint recognitions in this paper. After the 3D positions were obtained, many miss recognized points are occurred due to limitation of machine learning model. Hence, the post-processing algorithm is used to modify the signal of each 3D joint point on the sequence. To verify our model, 3 industrial cameras were installed in the baseball field in NTUS. To calibrate the camera, the intrinsic orientation of the camera can be calculated by the 2D single plane checkerboard, and the relative orientation of the camera can be calculated by a 3D cube. The k-means clustering algorithm is used to automatically select appropriate checkerboard from the image to prevent over-fitting of the intrinsic orientation of camera. K-fold is used to cross-validate relative orientation in multi-camera system. In a multi-camera system, the position and number of cameras in space will determine the accuracy of the 3D scene. In the 3D signal, dropping out the wrong feature points can make the signal closer to the real signal. This paper uses the confidence score (cs) of 2D human pose and anomaly detection (nd) to determine the wrong feature points. In signal processing, the data fitted by the Fourier filter and polynomials on the time sequence can be reduced data error.“human_B20” is a right pitcher dataset. For the joint points of the right wrist, right elbow, right shoulder, left waist, left knee, and left ankle, the automated processing model achieves an average error of less than 15 mm.