立體數位影像相關法與深度學習系統整合應用於三維量測、姿態辨識、手臂控制

Abstract

本論文整合數位影像相關法光學量測技術(Digital image correlation, DIC)與深度學習應用於工業問題。數位影像相關法為非接觸式、非破壞全場域、跨尺度的量測技術，以完備的數學理論，計算影像數值追蹤表面特徵，取得特徵的位移、速度、加速度、縮放、變形等資訊，具有高精度、高靈敏的特性。深度學習則以統計、機率領域為基礎，使用數據訓練模型，進而調整參數，以實現任務目標，可歸納複雜問題，泛用性高。藉由結合兩種技術的優勢，開發立體人體姿態檢測與辨識系統、手勢控制機械手臂以及多物件即時追蹤量測。 在立體人體姿態檢測與辨識系統中，首先使用了MediaPipe BlazePose檢測人體姿態特徵點，接著搭配雙相機立體數位影像相關法，成功獲取立體人體姿態特徵精確座標，並與彩色深度相機(RGB-D Camera)量測結果進行詳細討論與比較，呈現雙相機建構的三維座標的穩定性，並收集良好量測的姿態數據進行深度學習模型的訓練，在靜態與動態動作中，有效辨識出人體動作進行分類，能夠應用在許多領域之中。 手勢控制機械手臂利用自行訓練的手勢辨識模型，建立客製化的手勢指令，讓人機操作更為直覺，提高機械手臂控制的效率。最後利用立體數位影像相關法量測機械手臂運行性能，呈現手勢控制機械手臂的效能。 多物件即時追蹤量測使用YOLOv5模型辨識多物件類別，搭配RGB-D相機量測。提出在可視化深度影像中進行影像處理，提取物件形心與重心等資訊，有效提升追蹤的穩定性，使得追蹤不受物件光源、顏色影響量測精度。

This paper integrates digital image correlation (DIC), a technique used for optical measurement, and deep learning for industrial applications. Digital image correlation is a non-contact, non-destructive, full-field, and multiscale measurement technique that employs comprehensive mathematical theories to compute the displacements, velocities, accelerations, scaling, and deformations of surface features from images, offering high precision and sensitivity. Deep learning, on the other hand, is based on statistics and probability and utilizes data to train models and adjust parameters to achieve specific tasks. It can effectively handle complex problems and exhibits high generality. By combining the advantages of these two technologies, a system for stereoscopic human posture detection and recognition, gesture-controlled robotic arms, and real-time multi-object tracking and measurement are developed. In the stereoscopic human posture detection and recognition system, MediaPipe BlazePose is first used to detect human pose landmarks, which are then combined with stereo digital image correlation using dual cameras to accurately obtain the coordinates of stereoscopic human pose features. Detailed discussions and comparisons with measurements from a color-depth camera (RGB-D Camera) demonstrate the stability of the 3D coordinates constructed using dual cameras. Good-quality pose data is collected for training deep learning models, enabling effective recognition and classification of human movements in both static and dynamic actions. This system has potential applications in various fields. Gesture-controlled robotic arms utilize self-trained gesture recognition models to establish customized gesture commands, making human-machine interaction more intuitive and improving the efficiency of robotic arm control. Finally, the performance of the gesture-controlled robotic arm is measured using stereoscopic digital image correlation, showcasing the effectiveness of gesture control. For real-time multi-object tracking and measurement, the YOLOv5 model is employed to recognize multiple object classes, combined with RGB-D camera measurements. An image processing technique is proposed to extract object centroids and centers of gravity from visualized depth images, effectively enhancing tracking stability and mitigating the impact of object lighting and color on measurement accuracy.