Follow control of upper limb rehabilitation training based on Kinect and NAO robot_Journal of Biomedical Engineering

Authors：

LIU Xiaoguang ^1,2 , LI Simin ^1,2 , LIANG Tie ^1,2 , LI Jun ^1,2 , LOU Cunguang ^1,2 , WANG Hongrui ^1,2 ,  LIU Xiuling ^1,2

1. College of Electronic and Information Engineering, Hebei University, Baoding, Hebei 071002, P. R. China;
2. Key Laboratory of Digital Medical Engineering of Hebei Province, Baoding, Hebei 071002, P. R. China;

Corresponding author：

LIU Xiuling, Email: liuxiuling121@hotmail.com

Keywords：

Kinect; Follow control; NAO; Inverse kinematics; Median filter

DOI：

10.7507/1001-5515.202111009

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Gesture imitation is a common rehabilitation strategy in limb rehabilitation training. In traditional rehabilitation training, patients need to complete training actions under the guidance of rehabilitation physicians. However, due to the limited resources of the hospital, it cannot meet the training and guidance needs of all patients. In this paper, we proposed a following control method based on Kinect and NAO robot for the gesture imitation task in rehabilitation training. The method realized the joint angles mapping from Kinect coordination to NAO robot coordination through inverse kinematics algorithm. Aiming at the deflection angle estimation problem of the elbow joint, a virtual space plane was constructed and realized the accurate estimation of deflection angle. Finally, a comparative experiment for deflection angle of the elbow joint angle was conducted. The experimental results showed that the root mean square error of the angle estimation value of this method in right elbow transverse deflection and vertical deflection directions was 2.734° and 2.159°, respectively. It demonstrates that the method can follow the human movement in real time and stably using the NAO robot to show the rehabilitation training program for patients.

Citation： LIU Xiaoguang, LI Simin, LIANG Tie, LI Jun, LOU Cunguang, WANG Hongrui, LIU Xiuling. Follow control of upper limb rehabilitation training based on Kinect and NAO robot. Journal of Biomedical Engineering, 2022, 39(6): 1189-1198. doi: 10.7507/1001-5515.202111009 Copy

1.	王嘉津, 左国坤, 张佳楫, 等. 腕功能康复机器人按需辅助控制策略研究. 生物医学工程学杂志, 2020, 37(1): 129-135.
2.	程洪, 黄瑞, 邱静, 等. 康复机器人及其临床应用综述. 机器人, 2021, 43(5): 606-619.
3.	Afsar S I, Mirzayev I, Yemisci O U, et al. Virtual reality in upper extremity rehabilitation of stroke patients: a randomized controlled trial. J Stroke Cerebrovasc Dis, 2018, 27(12): 3473-3478.
4.	Schaham N G, Zeilig G, Weingarden H, et al. Game analysis and clinical use of the Xbox-Kinect for stroke rehabilitation. Int J Rehabil Res, 2018, 41(4): 323-330.
5.	Sanchez L, Courtine A, Gerard-Flavian G, et al. The use of a social assistive robot: NAO for post strokes rehabilitation therapy: a preliminary study. Comput Methods Biomech Biomed Eng, 2019, 22(Sup1): S478-S480.
6.	Bakhti K K A, Laffont I, Muthalib M, et al. Kinect-based assessment of proximal arm non-use after a stroke. J Neuroeng Rehabil, 2018, 15(1): 1-12.
7.	Semblantes P A, Andaluz V H, Lagla J, et al. Visual feedback framework for rehabilitation of stroke patients. Inform Med Unlocked, 2018, 13: 41-50.
8.	Dubey V N, Manna S K. Design of a game-based rehabilitation system using kinect sensor// 2019 Design of Medical Devices Conference. Minneapolis: ASME, 2019: V001T03A005.
9.	Cai L, Ma Y, Xiong S, et al. Validity and reliability of upper limb functional assessment using the microsoft Kinect V2 sensor. Appl Bionics Biomech, 2019, 2019: 7175240.
10.	侯增广, 赵新刚, 程龙, 等. 康复机器人与智能辅助系统的研究进展. 自动化学报, 2016, 42(12): 1765-1779.
11.	吴宏健, 李莉娜, 李龙, 等. 脑卒中后手功能康复机器人综合干预研究进展. 生物医学工程学杂志, 2019, 36(1): 151-156.
12.	Yavşan E, Uçar A. Gesture imitation and recognition using Kinect sensor and extreme learning machines. Measurement, 2016, 94: 852-861.
13.	Elbasiony R, Gomaa W. Humanoids skill learning based on real-time human motion imitation using Kinect. Intel Serv Robot, 2018, 11(2): 149-169.
14.	Hu N, Zheng L. Human action imitation system based on NAO robot// 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Chengdu: IEEE, 2019: 2261-2264.
15.	Wei Y, Zhao J. Designing human-like behaviors for anthropomorphic arm in humanoid robot NAO. Robotica, 2020, 38(7): 1205-1226.
16.	黄玲涛, 王彬, 倪涛, 等. 基于Kinect的机器人抓取系统研究. 农业机械学报, 2019, 50(1): 390-399.
17.	于建均, 门玉森, 阮晓钢, 等. 基于Kinect的NAO机器人动作模仿系统的研究与实现. 智能系统学报, 2016, 11(2): 180-187.
18.	滑少扬. 基于Kinect的NAO机器人的模仿动作的研究. 秦皇岛: 燕山大学, 2017.
19.	Zhang Z, Niu Y, Kong L, et al. A real-time upper-body robot imitation system. Int J Robot Control, 2019, 2(1): 49.
20.	Wang Z, Liang R, Chen Z, et al. Fast and intuitive kinematics mapping for human-robot motion imitating: A virtual-joint-based approach. IFAC-PapersOnLine, 2020, 53(2): 10011-10018.
21.	Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Kinect controlled NAO robot for telerehabilitation. J Intell Syst, 2020, 30(1): 224-239.
22.	Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Robot sensor system for supervised rehabilitation with real-time feedback. Multimed Tools Appl, 2020, 79(3): 26643-26660.
23.	冷舒, 吴克, 居鹤华. 机械臂运动学建模及解算方法综述. 宇航学报, 2019, 40(11): 1262-1273.
24.	马丛俊, 赵涛, 向国菲, 等. 基于逆运动学的柔性机械臂末端定位控制. 机械工程学报, 2021, 57(13): 163-171.
25.	Zhang T, Song Y, Wu H, et al. A novel method to identify DH parameters of the rigid serial-link robot based on a geometry model. Ind Robot, 2020, 48(1): 157-167.
26.	Liu X, Qiu C, Zeng Q, et al. Kinematics analysis and trajectory planning of collaborative welding robot with multiple manipulators. Procedia CIRP, 2019, 81: 1034-1039.
27.	Alam M M, Ibaraki S, Fukuda K. Kinematic modeling of six-axis industrial robot and its parameter identification: a tutorial. Int J Auto Tech-Kor, 2021, 15(5): 599-610.
28.	Xu N, Peng X, Peng L, et al. Modeling and kinematics analysis of a novel 5-DOF upper limb exoskeleton rehabilitation robot// 2020 39th Chinese Control Conference (CCC). Shenyang: IEEE, 2020: 1052-1057.
29.	Guo F, Cai H, Ceccarelli M, et al. Enhanced D-H: an improved convention for establishing a robot link coordinate system fixed on the joint. Ind Robot, 2019, 47(2): 197-205.
30.	Guida R, De Simone M C, Dašić P, et al. Modeling techniques for kinematic analysis of a six-axis robotic arm// IOP Conference Series: Materials Science and Engineering. Wuhan: IOP, 2019, 568(1): 012115.
31.	胡奎, 张继文, 董云飞, 等. 针对关节限位优化的7自由度机械臂逆运动学解法. 清华大学学报(自然科学版), 2020, 60(12): 1007-1015.
32.	周书华, 张文辉, 闻志, 等. 直角坐标机器人基于D-H参数的运动学建模与轨迹规划. 电工技术, 2020(24): 78-80, 83.
33.	Tölgyessy M, Dekan M, Chovanec Ľ, et al. Evaluation of the azure Kinect and its comparison to Kinect V1 and Kinect V2. Sensors, 2021, 21(2): 413.
34.	Assad-Uz-Zaman M, Islam M R, Rahman M H. Upper-extremity rehabilitation with NAO robot// 5th International Conference of Control, Dynamic Systems, and Robotics. Niagara Falls: CDSR, 2018: 117.
35.	刘晓光, 李梦楠, 王立玲, 等. 基于表面肌电信号的康复过程中肌疲劳有效性分析. 生物医学工程学杂志, 2019, 36(1): 80-84.

1. 王嘉津, 左国坤, 张佳楫, 等. 腕功能康复机器人按需辅助控制策略研究. 生物医学工程学杂志, 2020, 37(1): 129-135.
2. 程洪, 黄瑞, 邱静, 等. 康复机器人及其临床应用综述. 机器人, 2021, 43(5): 606-619.
3. Afsar S I, Mirzayev I, Yemisci O U, et al. Virtual reality in upper extremity rehabilitation of stroke patients: a randomized controlled trial. J Stroke Cerebrovasc Dis, 2018, 27(12): 3473-3478.
4. Schaham N G, Zeilig G, Weingarden H, et al. Game analysis and clinical use of the Xbox-Kinect for stroke rehabilitation. Int J Rehabil Res, 2018, 41(4): 323-330.
5. Sanchez L, Courtine A, Gerard-Flavian G, et al. The use of a social assistive robot: NAO for post strokes rehabilitation therapy: a preliminary study. Comput Methods Biomech Biomed Eng, 2019, 22(Sup1): S478-S480.
6. Bakhti K K A, Laffont I, Muthalib M, et al. Kinect-based assessment of proximal arm non-use after a stroke. J Neuroeng Rehabil, 2018, 15(1): 1-12.
7. Semblantes P A, Andaluz V H, Lagla J, et al. Visual feedback framework for rehabilitation of stroke patients. Inform Med Unlocked, 2018, 13: 41-50.
8. Dubey V N, Manna S K. Design of a game-based rehabilitation system using kinect sensor// 2019 Design of Medical Devices Conference. Minneapolis: ASME, 2019: V001T03A005.
9. Cai L, Ma Y, Xiong S, et al. Validity and reliability of upper limb functional assessment using the microsoft Kinect V2 sensor. Appl Bionics Biomech, 2019, 2019: 7175240.
10. 侯增广, 赵新刚, 程龙, 等. 康复机器人与智能辅助系统的研究进展. 自动化学报, 2016, 42(12): 1765-1779.
11. 吴宏健, 李莉娜, 李龙, 等. 脑卒中后手功能康复机器人综合干预研究进展. 生物医学工程学杂志, 2019, 36(1): 151-156.
12. Yavşan E, Uçar A. Gesture imitation and recognition using Kinect sensor and extreme learning machines. Measurement, 2016, 94: 852-861.
13. Elbasiony R, Gomaa W. Humanoids skill learning based on real-time human motion imitation using Kinect. Intel Serv Robot, 2018, 11(2): 149-169.
14. Hu N, Zheng L. Human action imitation system based on NAO robot// 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Chengdu: IEEE, 2019: 2261-2264.
15. Wei Y, Zhao J. Designing human-like behaviors for anthropomorphic arm in humanoid robot NAO. Robotica, 2020, 38(7): 1205-1226.
16. 黄玲涛, 王彬, 倪涛, 等. 基于Kinect的机器人抓取系统研究. 农业机械学报, 2019, 50(1): 390-399.
17. 于建均, 门玉森, 阮晓钢, 等. 基于Kinect的NAO机器人动作模仿系统的研究与实现. 智能系统学报, 2016, 11(2): 180-187.
18. 滑少扬. 基于Kinect的NAO机器人的模仿动作的研究. 秦皇岛: 燕山大学, 2017.
19. Zhang Z, Niu Y, Kong L, et al. A real-time upper-body robot imitation system. Int J Robot Control, 2019, 2(1): 49.
20. Wang Z, Liang R, Chen Z, et al. Fast and intuitive kinematics mapping for human-robot motion imitating: A virtual-joint-based approach. IFAC-PapersOnLine, 2020, 53(2): 10011-10018.
21. Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Kinect controlled NAO robot for telerehabilitation. J Intell Syst, 2020, 30(1): 224-239.
22. Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Robot sensor system for supervised rehabilitation with real-time feedback. Multimed Tools Appl, 2020, 79(3): 26643-26660.
23. 冷舒, 吴克, 居鹤华. 机械臂运动学建模及解算方法综述. 宇航学报, 2019, 40(11): 1262-1273.
24. 马丛俊, 赵涛, 向国菲, 等. 基于逆运动学的柔性机械臂末端定位控制. 机械工程学报, 2021, 57(13): 163-171.
25. Zhang T, Song Y, Wu H, et al. A novel method to identify DH parameters of the rigid serial-link robot based on a geometry model. Ind Robot, 2020, 48(1): 157-167.
26. Liu X, Qiu C, Zeng Q, et al. Kinematics analysis and trajectory planning of collaborative welding robot with multiple manipulators. Procedia CIRP, 2019, 81: 1034-1039.
27. Alam M M, Ibaraki S, Fukuda K. Kinematic modeling of six-axis industrial robot and its parameter identification: a tutorial. Int J Auto Tech-Kor, 2021, 15(5): 599-610.
28. Xu N, Peng X, Peng L, et al. Modeling and kinematics analysis of a novel 5-DOF upper limb exoskeleton rehabilitation robot// 2020 39th Chinese Control Conference (CCC). Shenyang: IEEE, 2020: 1052-1057.
29. Guo F, Cai H, Ceccarelli M, et al. Enhanced D-H: an improved convention for establishing a robot link coordinate system fixed on the joint. Ind Robot, 2019, 47(2): 197-205.
30. Guida R, De Simone M C, Dašić P, et al. Modeling techniques for kinematic analysis of a six-axis robotic arm// IOP Conference Series: Materials Science and Engineering. Wuhan: IOP, 2019, 568(1): 012115.
31. 胡奎, 张继文, 董云飞, 等. 针对关节限位优化的7自由度机械臂逆运动学解法. 清华大学学报(自然科学版), 2020, 60(12): 1007-1015.
32. 周书华, 张文辉, 闻志, 等. 直角坐标机器人基于D-H参数的运动学建模与轨迹规划. 电工技术, 2020(24): 78-80, 83.
33. Tölgyessy M, Dekan M, Chovanec Ľ, et al. Evaluation of the azure Kinect and its comparison to Kinect V1 and Kinect V2. Sensors, 2021, 21(2): 413.
34. Assad-Uz-Zaman M, Islam M R, Rahman M H. Upper-extremity rehabilitation with NAO robot// 5th International Conference of Control, Dynamic Systems, and Robotics. Niagara Falls: CDSR, 2018: 117.
35. 刘晓光, 李梦楠, 王立玲, 等. 基于表面肌电信号的康复过程中肌疲劳有效性分析. 生物医学工程学杂志, 2019, 36(1): 80-84.

Journal of Biomedical Engineering

Follow control of upper limb rehabilitation training based on Kinect and NAO robot

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content