請用此 Handle URI 來引用此文件:
http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/84878
完整後設資料紀錄
DC 欄位 | 值 | 語言 |
---|---|---|
dc.contributor.advisor | 黃漢邦(Han-Pang Huang) | |
dc.contributor.author | Ze-Feng Zhan | en |
dc.contributor.author | 詹澤鋒 | zh_TW |
dc.date.accessioned | 2023-03-19T22:30:27Z | - |
dc.date.copyright | 2022-10-19 | |
dc.date.issued | 2022 | |
dc.date.submitted | 2022-08-29 | |
dc.identifier.citation | [1] Yang Chengfu and Zheng Manjin's Masterpiece [Yang Style Zhengzi Tai-Chi 37 Styles]. Youtube. 2022, from https://www.youtube.com/watch?v=kBI9qOFliTU&ab_channel=LeeChanTaichiTV [2] M. Alibeigi, S. Rabiee, and M. N. Ahmadabadi, 'Inverse Kinematics Based Human Mimicking System Using Skeletal Tracking Technology,' Journal of Intelligent & Robotic Systems, Vol. 85, No. 1, pp. 27-45, 2017. [3] M. Andtfolk, L. Nyholm, H. Eide, and L. Fagerström, 'Humanoid Robots in the Care of Older Persons: A Scoping Review,' Assistive Technology, Vol. 1, No. 1, pp. 1-9, 2021. [4] L. Biagiotti and C. Melchiorri, Trajectory Planning for Automatic Machines and Robots, 1st edition, New York: Springer Science & Business Media, 2008. [5] C. Breazeal, C. D. Kidd, A. L. Thomaz, G. Hoffman, and M. Berlin, 'Effects of Nonverbal Communication on Efficiency and Robustness in Human-Robot Teamwork,' Proc. of 2005 IEEE/RSJ international conference on intelligent robots and systems, Edmonton, AB, Canada, pp. 708-713, 2005. [6] T. Breuer, G. R. Giorgana Macedo, R. Hartanto, N. Hochgeschwender, D. Holz, F. Hegger, Z. Jin, C. Müller, J. Paulus, and M. Reckhaus, 'Johnny: An Autonomous Service Robot for Domestic Environments,' Journal of intelligent & robotic systems, Vol. 66, No. 1, pp. 245-272, 2012. [7] S. R. Buss, 'Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares Methods,' IEEE Journal of Robotics and Automation, Vol. 17, No. 1-19, pp. 16, 2004. [8] J. H. Chen, 'Optimal Contact Wrench Controller for Humanoid Robots Based on Floating Base Kinematics,' Master Thesis, Department of Mechanical Engineering, National Taiwan University, Taipei, Taiwan, 2016. [9] Y. T. Chen, 'Smooth Movement Sequence Generation Based on Music Analy-Sis for Humanoid Robots,' Master Thesis, Department of Mechanical Engineering, National Taiwan University, 2019. [10] R. Cisneros, S. i. Nakaoka, M. Morisawa, K. Kaneko, S. Kajita, T. Sakaguchi, and F. Kanehiro, 'Effective Teleoperated Manipulation for Humanoid Robots in Partially Unknown Real Environments: Team Aist-Nedo’s Approach for Performing the Plug Task During the Drc Finals,' Advanced Robotics, Vol. 30, No. 24, pp. 1544-1558, 2016. [11] S. Filiatrault and A.-M. Cretu, 'Human Arm Motion Imitation by a Humanoid Robot,' Proc. of 2014 IEEE International Symposium on Robotic and Sensors Environments (ROSE) Proceedings, Timisoara, Romania, pp. 31-36, 2014. [12] L. Gong, B. Chen, W. Xu, C. Liu, X. Li, Z. Zhao, and L. Zhao, 'Motion Similarity Evaluation between Human and a Tri-Co Robot During Real-Time Imitation with a Trajectory Dynamic Time Warping Model,' Sensors, Vol. 22, No. 5, pp. 1968, 2022. [13] H.-K. Hsu, H.-P. Huang, and S.-Y. Lo, 'A Humanoid Robotics Simulation and Control Platform for Nino,' International Journal of the Digital Human, Vol. 1, No. 2, pp. 169-194, 2016. [14] H. K. Hsu, 'Energy-Saving Walking Pattern Generation, Multi-Task Whole-Body Control and Safety for Humanoid Robots,' Doctral Dissertation, Department of Mechanical Engineering, National Taiwan University, Taipei, Taiwan, pp. 1-160, 2020. [15] H.-P. Huang, J.-L. Yan, and T.-H. Cheng, 'State-Incremental Optimal Control of 3d Cog Pattern Generation for Humanoid Robots,' Advanced Robotics, Vol. 27, No. 3, pp. 175-188, 2013. [16] C.-L. Hwang and G.-H. Liao, 'Real-Time Pose Imitation by Mid-Size Humanoid Robot with Servo-Cradle-Head Rgb-D Vision System,' IEEE Transactions on Systems, Man, and Cybernetics: Systems, Vol. 49, No. 1, pp. 181-191, 2018. [17] S. Kajita, F. Kanehiro, K. Kaneko, K. Yokoi, and H. Hirukawa, 'The 3d Linear Inverted Pendulum Mode: A Simple Modeling for a Biped Walking Pattern Generation,' Proc. of Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No. 01CH37180), Vol. 1, pp. 239-246, 2001. [18] S. Kajita, M. Morisawa, K. Harada, K. Kaneko, F. Kanehiro, K. Fujiwara, and H. Hirukawa, 'Biped Walking Pattern Generator Allowing Auxiliary Zmp Control,' Proc. of 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, Vol. 1, No. 1, pp. 2993-2999, 2006. [19] A. Karar, S. Said, and T. Beyrouthy, 'Pepper Humanoid Robot as a Service Robot: A Customer Approach,' Proc. of 2019 3rd International Conference on Bio-engineering for Smart Technologies (BioSMART), Paris, France, pp. 1-4, 2019. [20] H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, and E. Osawa, 'Robocup: The Robot World Cup Initiative,' Proc. of Proceedings of the first international conference on Autonomous agents, pp. 340-347, 1997. [21] J. Koenemann, F. Burget, and M. Bennewitz, 'Real-Time Imitation of Human Whole-Body Motions by Humanoids,' Proc. of 2014 IEEE International Conference on Robotics and Automation (ICRA), Marina Del Rey, CA USA, pp. 2806-2812, 2014. [22] M. Koptev, N. Figueroa, and A. Billard, 'Real-Time Self-Collision Avoidance in Joint Space for Humanoid Robots,' IEEE Robotics and Automation Letters, Vol. 6, No. 2, pp. 1240-1247, 2021. [23] H. Lee, S. D. Kim, and M. A. U. Al Amin, 'Control Framework for Collaborative Robot Using Imitation Learning-Based Teleoperation from Human Digital Twin to Robot Digital Twin,' Mechatronics, Vol. 85, No. 1, pp. 102833, 2022. [24] J.-H. Lee, 'Full-Body Imitation of Human Motions with Kinect and Heterogeneous Kinematic Structure of Humanoid Robot,' Proc. of 2012 IEEE/SICE International Symposium on System Integration (SII), Fukuoka, Japan, pp. 93-98, 2012. [25] J. Lei, M. Song, Z.-N. Li, and C. Chen, 'Whole-Body Humanoid Robot Imitation with Pose Similarity Evaluation,' Signal Processing, Vol. 108, No. 1, pp. 136-146, 2015. [26] Q. Li, A. Takanishi, and I. Kato, 'A Biped Walking Robot Having a Zmp Measurement System Using Universal Force-Moment Sensors,' Proc. of Proceedings IROS'91: IEEE/RSJ International Workshop on Intelligent Robots and Systems' 91, Osaka, Japan, pp. 1568-1573, 1991. [27] S.-R. Lu, 'Implementation of Pre-Engagement Detection on Human-Robot Interaction in Complex Environments,' Master Thesis, Department of Mechanical Engineering, National Taiwan University, Taipei, Taiwan, 2020. [28] S. Lu, 'Real-Time Humanoid Robot Dance System Based on Music Genre Classification,' Master Thesis, Department of Mechanical Engineering, National Taiwan University, Taipei, Taiwan, 2021. [29] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, and J. Lee, 'Mediapipe: A Framework for Building Perception Pipelines,' arXiv preprint arXiv:1906.08172,2019. [30] R. C. Luo, B.-H. Shih, and T.-W. Lin, 'Real Time Human Motion Imitation of Anthropomorphic Dual Arm Robot Based on Cartesian Impedance Control,' Proc. of 2013 IEEE international symposium on robotic and sensors environments (ROSE), Washington, DC, USA, pp. 25-30, 2013. [31] J. Martinez, R. Hossain, J. Romero, and J. J. Little, 'A Simple yet Effective Baseline for 3d Human Pose Estimation,' Proc. of Proceedings of the IEEE international conference on computer vision, Venice, Italy, pp. 2640-2649, 2017. [32] O. Mazhar, B. Navarro, S. Ramdani, R. Passama, and A. Cherubini, 'A Real-Time Human-Robot Interaction Framework with Robust Background Invariant Hand Gesture Detection,' Robotics and Computer-Integrated Manufacturing, Vol. 60, No. 1, pp. 34-48, 2019. [33] M. Mistry, J. Nakanishi, G. Cheng, and S. Schaal, 'Inverse Kinematics with Floating Base and Constraints for Full Body Humanoid Robot Control,' Proc. of Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots, Daejeon, South Korea, pp. 22-27, 2008. [34] M. Mori, K. F. MacDorman, and N. Kageki, 'The Uncanny Valley [from the Field],' IEEE Robotics & Automation Magazine, Vol. 19, No. 2, pp. 98-100, 2012. [35] S. Mukherjee, D. Paramkusam, and S. K. Dwivedy, 'Inverse Kinematics of a Nao Humanoid Robot Using Kinect to Track and Imitate Human Motion,' Proc. of 2015 International Conference on Robotics, Automation, Control and Embedded Systems (RACE), Chennai, India, pp. 1-7, 2015. [36] N. Naksuk, C. G. Lee, and S. Rietdyk, 'Whole-Body Human-to-Humanoid Motion Transfer,' Proc. of 5th IEEE-RAS International Conference on Humanoid Robots, 2005., Tsukuba, Japan, pp. 104-109, 2005. [37] K. Narukawa, T. Yoshiike, K. Tanaka, and M. Kuroda, 'Real-Time Collision Detection Based on One Class Svm for Safe Movement of Humanoid Robot,' Proc. of 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), Birmingham, UK, pp. 791-796, 2017. [38] S. B. Niku, Introduction to Robotics: Analysis, Control, Applications, 2 nd edition, New York: John Wiley & Sons, 2020. [39] Y. Nishida, T. Sonoda, and K. Ishii, 'Jacobian Matrix Derived from Cross Product and Its Application into High Power Joint Mechanism Analysis,' Journal of Bionic Engineering, Vol. 7, No. 1, pp. S218-S223, 2010. [40] Y. Ou, J. Hu, Z. Wang, Y. Fu, X. Wu, and X. Li, 'A Real-Time Human Imitation System Using Kinect,' International Journal of Social Robotics, Vol. 7, No. 5, pp. 587-600, 2015. [41] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli, '3d Human Pose Estimation in Video with Temporal Convolutions and Semi-Supervised Training,' Proc. of Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, pp. 7753-7762, 2019. [42] L. Penco, B. Clément, V. Modugno, E. M. Hoffman, G. Nava, D. Pucci, N. G. Tsagarakis, J.-B. Mouret, and S. Ivaldi, 'Robust Real-Time Whole-Body Motion Retargeting from Human to Humanoid,' Proc. of 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), Beijing, China, pp. 425-432, 2018. [43] O. Pino, G. Palestra, R. Trevino, and B. De Carolis, 'The Humanoid Robot Nao as Trainer in a Memory Program for Elderly People with Mild Cognitive Impairment,' International Journal of Social Robotics, Vol. 12, No. 1, pp. 21-33, 2020. [44] J. J. Quiroz-Omaña and B. V. Adorno, 'Whole-Body Control with (Self) Collision Avoidance Using Vector Field Inequalities,' IEEE Robotics and Automation Letters, Vol. 4, No. 4, pp. 4048-4053, 2019. [45] S. Saeedvand, M. Jafari, H. S. Aghdasi, and J. Baltes, 'A Comprehensive Survey on Humanoid Robot Development,' The Knowledge Engineering Review, Vol. 34, No. 1,2019. [46] S. K. Saha, 'A Unified Approach to Space Robot Kinematics,' IEEE transactions on robotics and automation, Vol. 12, No. 3, pp. 401-405, 1996. [47] S. Schiffer, A. Ferrein, and G. Lakemeyer, 'Caesar: An Intelligent Domestic Service Robot,' Intelligent Service Robotics, Vol. 5, No. 4, pp. 259-273, 2012. [48] L. Sentis and O. Khatib, 'A Whole-Body Control Framework for Humanoids Operating in Human Environments,' Proc. of Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., Orlando, FL, USA, pp. 2641-2648, 2006. [49] T. B. Sheridan, 'Human–Robot Interaction: Status and Challenges,' Human factors, Vol. 58, No. 4, pp. 525-532, 2016. [50] A. Taheri, M. Alemi, A. Meghdari, H. Pouretemad, and S. Holderread, 'Clinical Application of Humanoid Robots in Playing Imitation Games for Autistic Children in Iran,' Procedia-Social and Behavioral Sciences, Vol. 176, No. 1, pp. 898-906, 2015. [51] T. Tao, Z. Zhang, and X. Yang, 'Visual Perception Method Based on Human Pose Estimation for Humanoid Robot Imitating Human Motions,' Proc. of 2021 2nd International Conference on Control, Robotics and Intelligent System, Qingdao China, pp. 54-61, 2021. [52] L.-W. Tsai, Robot Analysis: The Mechanics of Serial and Parallel Manipulators, 1st edition, New York: John Wiley & Sons, 1999. [53] M. Vukobratović and B. Borovac, 'Zero-Moment Point—Thirty Five Years of Its Life,' International journal of humanoid robotics, Vol. 1, No. 1, pp. 157-173, 2004. [54] M. Vukobratović, B. Borovac, and V. Potkonjak, 'Zmp: A Review of Some Basic Misunderstandings,' International Journal of Humanoid Robotics, Vol. 3, No. 02, pp. 153-175, 2006. [55] F. Wang, C. Tang, Y. Ou, and Y. Xu, 'A Real-Time Human Imitation System,' Proc. of Proceedings of the 10th World Congress on Intelligent Control and Automation, Beijing, China, pp. 3692-3697, 2012. [56] J. Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, and L. Shao, 'Deep 3d Human Pose Estimation: A Review,' Computer Vision and Image Understanding, Vol. 210, No. 1, pp. 103225, 2021. [57] S. Wang, P. Yang, and H. Sun, 'Design and Implementation of Auditory System for Mobile Robot Based on Kinect Sensor,' Proc. of 2016 12th World Congress on Intelligent Control and Automation (WCICA), Guilin, China, pp. 3094-3098, 2016. [58] Y. H. Wang, 'Synchronized Leg-Arm Motion Planning and Whole-Body Momentum Control for Humanoid Robots ' Master Thesis, Department of Mechanical Engineering, National Taiwan University, Taipei, Taiwan, 2018. [59] Z. Wang, X. Yuan, and C. Zhang, 'Design and Implementation of Humanoid Robot Behavior Imitation System Based on Skeleton Tracking,' Proc. of 2019 Chinese Control And Decision Conference (CCDC), Nanchang, China, pp. 3541-3546, 2019. [60] W. T. Weng, 'Human-Robot Interaction on Dual-Arm Mobile Robot Robot with Hand Gesture Recognition and Visual Servoing,' Master Thesis, Department of Mechanical Engineering, National Taiwan University, Taipei, Taiwan, 2021. [61] J.-L. Yan and H.-P. Huang, 'A Fast and Smooth Walking Pattern Generator of Biped Robot Using Jacobian Inverse Kinematics,' Proc. of 2007 IEEE Workshop on Advanced Robotics and Its Social Impacts, Hsinchu, Taiwan, pp. 1-6, 2007. [62] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, 'Mediapipe Hands: On-Device Real-Time Hand Tracking,' arXiv preprint arXiv:2006.10214,2020. [63] L. Zhang, Z. Cheng, Y. Gan, G. Zhu, P. Shen, and J. Song, 'Fast Human Whole Body Motion Imitation Algorithm for Humanoid Robots,' Proc. of 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), Qingdao, China, pp. 1430-1435, 2016. | |
dc.identifier.uri | http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/84878 | - |
dc.description.abstract | 針對人形機器人的複雜全身動作規劃,本論文提出一個以模仿影片中人類動作來規劃近似人類動作的機器人動作的系統,稱作模仿系統。並且為我們的人形機器人搭建一套互動系統,使其有能力進行基本的人機互動。 在模仿系統中,首先要抓取模仿的人類目標的動作,我們利用3D人體關鍵點預測模型,取得影片中人體的關鍵點的3D座標。接下來透過本論文提出的映射方法,其中參考了機器人的控制方法和穩定性,將關鍵點3D座標轉換並得到機器人執行該動作所需的各種軌跡檔案。此外,我們提出幾種後處理方法去處理生成的軌跡。 而互動系統中,我們使用透過模仿系統得到的動作資料庫來規劃動作,並且建立了視覺和語音的系統,使機器人能辨識人類手勢或是姿勢、可以跟人進行對話,並接收語音資訊進行動作。另外也搭載了學長建立的音樂節奏辨識系統,能讓機器人跟上音樂節拍跳舞。這個系統使我們的人形機器人能和人互動。 最後我們透過上述系統,完成幾項簡單的人機互動場景,證明模仿系統動作規劃的方便快速及有效,以及我們的互動系統的完備。 | zh_TW |
dc.description.abstract | In this thesis, we propose an imitation system that imitates human motions in videos to plan robot actions that are similar to human motions, with the aim of the complicated whole-body action planning of humanoid robots. Additionally, we created an interaction system that will enable basic human-robot interaction for our humanoid robot. To obtain the 3D coordinates of the key points on the human body, we used the 3D pose estimation model. The key points were then transformed to various trajectory files needed by the robot to complete the motion, using the mapping method proposed in this research, which refers to the control strategy and stability of the robot. In addition, we proposed some post-processing method to post-process the trajectories. In the interaction system, we created a speech and vision system so that the robot could detect human gestures or postures and converse with people. It also has a music rhythm recognition system developed by seniors that enables the robot to dance to the beats of the song. Finally, through this system, we completed several human-robot interaction scenarios, which proved the convenience, and effectiveness of motion planning with an imitation system, and the completeness of the interaction system. | en |
dc.description.provenance | Made available in DSpace on 2023-03-19T22:30:27Z (GMT). No. of bitstreams: 1 U0001-2608202218091700.pdf: 7645855 bytes, checksum: e7dfb71bc6ccf92fe5b2fd059bcc65f5 (MD5) Previous issue date: 2022 | en |
dc.description.tableofcontents | 誌謝 i 摘要 iii Abstract v List of Tables xi List of Figures xiii Chapter 1 Introduction 1 1.1 Motivations 1 1.2 Contributions 3 1.3 Organization 4 Chapter 2 Humanoid Robot System 7 2.1 Kinematics of Robots 8 2.1.1 Forward Kinematics 9 2.1.2 Inverse Kinematics 11 2.1.3 Trajectory Planning 13 2.2 Pattern Generation 15 2.2.1 Dynamic Model of Humanoid Robot 15 2.2.2 Linear Quadratic State-Incremental Control 18 2.3 Floating Based Kinematics 20 2.4 Summary 24 Chapter 3 Motion Imitation System 27 3.1 Related Works and the Proposed Imitation Method 28 3.1.1 Literature Review 28 3.1.2 Methods in this Thesis 32 3.2 Motion Captured by 3D Pose Estimation Model 35 3.2.1 Introduction 35 3.2.2 3D Pose Estimation Method 37 3.3 Human Motion to Humanoid Robot Motion 41 3.3.1 Upper Body Mapping Method 42 3.3.2 Lower Body Mapping Method and Balance Control 47 3.3.3 Mapping of the Other Part 53 3.4 Trajectory Post-Processing 55 3.4.1 Joint Limit and Collision Avoidance 56 3.4.2 Trajectory smoothing 65 3.4.3 Fine-Tuning for Better Stability 68 3.4.4 Motion Data 70 3.5 Summary 71 Chapter 4 Interaction System 73 4.1 Introduction 74 4.2 Online Motion Planning and Control 76 4.2.1 IK Controller 77 4.2.2 Online Planner 78 4.3 Vision System 79 4.3.1 Feature Extraction 80 4.3.2 Hand Gesture Recognition 81 4.4 Auditory System 84 4.4.1 Speech Recognition System 85 4.4.2 Dance with Music 86 4.5 Summary 87 Chapter 5 Simulations and Experiments 89 5.1 Specification of the NTU Humanoid Robot 89 5.1.1 Hardware 89 5.1.2 Software System 91 5.1.3 Simulation Environment 91 5.2 Simulation Results 92 5.2.1 The Details of the Imitation System and Interaction System 92 5.2.2 Collision Avoidance Results 96 5.2.3 Results of Gesture Recognition 99 5.2.4 The Performance of the Imitation System 100 5.2.5 Dancing with Music 102 5.2.6 Discussion 104 5.3 Experiment Scenarios and Results 105 5.3.1 Perform Motions Planned by Imitation System 105 5.3.2 Greeting to Users 106 5.3.3 Perform Dance to Users 108 5.3.4 Exercising with the User 110 5.3.5 Similarity Evaluation 115 5.3.6 Discussion 117 5.4 Summary 119 Chapter 6 Conclusions and Future Works 121 6.1 Conclusions 121 6.2 Future Works 122 References 125 | |
dc.language.iso | en | |
dc.title | 人形機器人的動作模仿系統及其應用 | zh_TW |
dc.title | Imitation System of Humanoid Robots and Its Applications | en |
dc.type | Thesis | |
dc.date.schoolyear | 110-2 | |
dc.description.degree | 碩士 | |
dc.contributor.oralexamcommittee | 郭重顯(Chung-Hsien Kuo),黃國勝(Kao-Shing Hwang),蔡清池(Ching-Chih Tsai),蔡清元(Tsing-Juan Tsay) | |
dc.subject.keyword | 人形機器人,模仿,3D關鍵點預測,人機互動,MediaPipe,動作規劃,動作資料庫, | zh_TW |
dc.subject.keyword | humanoid robot,imitation system,3D pose estimation,human-robot interaction,MediaPipe,motion planning,motion database, | en |
dc.relation.page | 130 | |
dc.identifier.doi | 10.6342/NTU202202870 | |
dc.rights.note | 同意授權(限校園內公開) | |
dc.date.accepted | 2022-08-29 | |
dc.contributor.author-college | 工學院 | zh_TW |
dc.contributor.author-dept | 機械工程學研究所 | zh_TW |
dc.date.embargo-lift | 2024-08-31 | - |
顯示於系所單位: | 機械工程學系 |
文件中的檔案:
檔案 | 大小 | 格式 | |
---|---|---|---|
U0001-2608202218091700.pdf 目前未授權公開取用 | 7.47 MB | Adobe PDF | 檢視/開啟 |
系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。