手語機器人之軌跡規劃與即時控制

Abstract

隨著智慧機器人技術的持續進步，其應用逐步從工業任務延伸至人機互動領域。而機器人與使用者之間的溝通能力亦為重要研究議題。雖然語音與文字已廣泛應用於人機界面設計，但對於以手語作為主要溝通媒介的聾人而言，這類方式未必合適。因此，本研究旨在開發一套具備台灣手語溝通能力的機器人系統，透過手語辨識、動作模仿與展示手語語句動作，實現跟聾人之間的互動能力。 在辨識系統方面，本研究採用視覺骨架分析技術，利用 Google MediaPipe 架構萃取影像中人物之關節資訊，以降低原始影像特徵的處理複雜度。搭配機器學習辨識模型，進行動態手語詞彙與句子的辨識訓練。訓練資料來自臺灣大學機器人實驗室所建立的台灣手語影片資料集以及公開的資料集，來進行模型的訓練與測試。最終，此機器學習的方法也在台灣手語辨識研究中表現出不錯的辨識率。 在手語動作生成與模仿方面，本研究提出一套整合式手語模仿與控制系統，將辨識結果轉換為手語人型機器人手臂手掌的控制指令。該系統透過角度映射與運動解算，實現多關節的協調控制。此外，亦設計自碰撞避免機制與軌跡後處理方法，以提升模仿動作的穩定性與流暢性。最終結合雙臂的規劃與控制系統，實現機器人的手語動作展示。 整體實驗結果表示本系統能準確呈現複雜的台灣手語動作，並具備基本的手語辨識能力，為未來手語機器人在輔助聾人溝通和手語教學等應用令譽奠定基礎。

With the continuous advancement of intelligent robotics technology, applications have gradually expanded from industrial tasks to the domain of human–robot interaction. Communication between robots and users has thus become a critical area of research. While voice and text-based interfaces are widely adopted in human–computer interaction, such methods may not be suitable for individuals who rely primarily on sign language. Therefore, this study aims to develop a humanoid robot system capable of communicating using Taiwanese Sign Language (TSL), enabling interaction with the deaf community through sign language recognition, motion imitation, and full-sentence sign expression. For the recognition component, this study adopts a visual skeleton-based analysis approach using Google MediaPipe to extract joint information from video frames, thereby reducing the complexity of processing raw image features. A machine learning model, trained on both a self-built TSL video dataset from the NTU Robotics Laboratory and publicly available datasets, is employed to recognize dynamic TSL words and sentences. The proposed approach demonstrates promising accuracy and efficiency in TSL recognition tasks. In terms of sign language generation and imitation, this research presents an integrated control system that translates recognition results into joint control commands for the robot’s arms and hands. By mapping joint angles and solving inverse kinematics, the system achieves coordinated multi-joint control. Additionally, a self-collision avoidance mechanism and post-processing methods are implemented to enhance the stability and smoothness of imitated motions. The system ultimately integrates dual-arm planning and control to enable the humanoid robot to perform complete TSL sentence gestures. Experimental results confirm that the proposed system can accurately reproduce complex TSL movements and possesses fundamental recognition capabilities. This work establishes a solid foundation for future applications of sign language robots in communication assistance, language education, and accessible human–robot interaction.