A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training_Journal of Biomedical Engineering

Authors：

XING Mengmeng , WEI Guohui , LIU Jing , ZHANG Junzhong , YANG Feng ,  CAO Hui

School of Science and technology, Shandong University of Traditional Chinese Medicine, Jinan 250355, P.R.China;

Corresponding author：

Keywords：

action capture; action feature extraction; deep learning; action recognition; orthopedic rehabilitation training

DOI：

10.7507/1001-5515.201906053

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Human motion recognition (HAR) is the technological base of intelligent medical treatment, sports training, video monitoring and many other fields, and it has been widely concerned by all walks of life. This paper summarized the progress and significance of HAR research, which includes two processes: action capture and action classification based on deep learning. Firstly, the paper introduced in detail three mainstream methods of action capture: video-based, depth camera-based and inertial sensor-based. The commonly used action data sets were also listed. Secondly, the realization of HAR based on deep learning was described in two aspects, including automatic feature extraction and multi-modal feature fusion. The realization of training monitoring and simulative training with HAR in orthopedic rehabilitation training was also introduced. Finally, it discussed precise motion capture and multi-modal feature fusion of HAR, as well as the key points and difficulties of HAR application in orthopedic rehabilitation training. This article summarized the above contents to quickly guide researchers to understand the current status of HAR research and its application in orthopedic rehabilitation training.

Citation： XING Mengmeng, WEI Guohui, LIU Jing, ZHANG Junzhong, YANG Feng, CAO Hui. A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training. Journal of Biomedical Engineering, 2020, 37(1): 174-178, 184. doi: 10.7507/1001-5515.201906053 Copy

1.	Guerra J, Uddin J, Nilsen D, et al. Capture, learning, and classification of upper extremity movement primitives in healthy controls and stroke patients//2017 International Conference on Rehabilitation Robotics (ICORR).lodon, UK: IEEE, 2017: 547-554.
2.	Wang Yufan, Zhao Yuliang, Chan R H, et al. Volleyball skill assessment using a single wearable micro inertial measurement unit at wrist. IEEE Access, 2018, 6: 13758-13765.
3.	潘月浩, 宋执环, 杜往泽, 等. 适用于老年公寓的动作识别设计方法. 电子测量与仪器学报, 2017, 31(1): 29-35.
4.	金刚, 李德华, 周学泳. 表演动画中的运动捕捉技术. 中国图象图形学报, 2000, 5(3): 88-91.
5.	朱红蕾, 朱昶胜, 徐志刚. 人体行为识别数据集研究进展. 自动化学报, 2018, 44(06): 978-1004.
6.	Alt Murphy M, Murphy S, Persson H C, et al. Kinematic analysis using 3D motion capture of drinking task in People with and without upper-extremity impairments. J Vis Exp, 2018(133): e57228.
7.	Callejas-Cuervo M, Gutierrez R M, Hernandez A I. Joint amplitude MEMS based measurement platform for low cost and high accessibility telerehabilitation: elbow case study. J Bodyw Mov Ther, 2017, 21(3): 574-581.
8.	Palumbo F, Gallicchio C, Pucci R, et al. Human activity recognition using multisensor data fusion based on reservoir computing. J Ambient Intell Smart Environ, 2016, 8(2): 87-107.
9.	Ryoo M S, Chen C C, Aggarwal J K, et al. An overview of contest on semantic description of human activities (SDHA) 2010//2010 International Conference on Pattern Recognition (ICPP).Berlin, Germany, 2010: 270-285.
10.	Wang Jiang, Liu Zicheng, Wu Ying, et al. Mining actionlet ensemble for action recognition with depth cameras//2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).RI, USA: IEEE, 2012: 1290-1297.
11.	Chen C, Jafari R, Kehtarnavaz N. UTD-MHAD: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor/2015 IEEE International Conference on Image Processing(ICIP).QC, Canada: IEEE, 2015: 168-172.
12.	王剑, 李春雨, 李跃新. 结合区域HOF和字典学习的人体行为识别方法. 计算机应用研究, 2017, 34(9): 2863-2866.
13.	肖玉玲. 结合HOG/HOF级联特征和多层分类器的人体行为识别. 计算机工程与设计, 2017, 38(9): 2567-2572.
14.	Yao Hongxian, Jiang Xinghao, Sun Tanfeng, et al. 3D human action recognition based on the spatial-temporal moving skeleton descriptor//2017 IEEE International Conference on Multimedia and Expo (ICME).Shenzhen, China: IEEE, 2017: 937-942.
15.	Shih T Y, Hsieh Y W, WU C Y, et al. Quantifying kinematic features of upper-extremity movement strategies during tooth brushing in patients with stroke and healthy controls. J Neurol Sci, 2017, 381: 995.
16.	Hassan M M, Uddin M Z, Mohamed A A. A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems, 2018, 81: 307-313.
17.	胡婷鸿, 万雷, 刘太昂, 等. 深度学习在图像识别及骨龄评估中的优势及应用前景. 法医学杂志, 2017, 33(6): 629-634, 639.
18.	Nweke H F, Teh Y W, Al-garadi M A, et al. Deep learning algorithms for human activity recognition using Mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl, 2018, 105: 233-261.
19.	张瑞, 李其申, 储珺. 基于3D卷积神经网络的人体动作识别算法. 计算机工程, 2019, 45(01): 259-263.
20.	Pham H H, Khoudour L, Crouzil A, et al. Exploiting deep residual networks for human action recognition from skeletal data. Computer Vision and Image Understanding, 2018, 170:51-66.
21.	Li Honggui, Trocan M. Deep learning of smartphone sensor data for personal health assistance. Microelectronics J, 2019, 88: 164-172.
22.	胡双杰, 秦建邦, 郭薇. 基于特征自动提取的跌倒检测算法. 传感技术学报, 2018, 31(12): 1842-1847.
23.	裴晓敏, 范慧杰, 唐延东. 时空特征融合深度学习网络人体行为识别方法. 红外与激光工程, 2018, 47(02): 55-60.
24.	Eitel A, Springenberg J T, Spinello L, et al. Multimodal deep learning for robust RGB-D object recognition//2015 IEEE International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany: IEEE, 2015: 681-687.
25.	Kim J H, Hong G S, Kim B G, et al. DeepGesture: deep learning-based gesture recognition scheme using motion sensors. Displays, 2018, 55: 38-45.
26.	李玉鹏, 刘婷婷, 张良. 基于深度学习的人体动作识别方法. 计算机应用研究. 2019, 37(1). DOI:10.19734/j.issn.1001-3695.2018.05.0499.
27.	罗会兰, 童康, 孔繁胜. 基于深度学习的视频中人体动作识别进展综述. 电子学报, 2019, 47(05): 1162-1173.
28.	Fernandez-Cervantes V, Neubauer N, Hunter B, et al. VirtualGym: A kinect-based system for seniors exercising at home. Entertain Comput, 2018, 27: 60-72.
29.	陈朝晖, 张文娣. 理筋正骨手法结合运动训练治疗慢性颈痛的临床观察. 中国中西医结合杂志, 2018, 38(11): 1327-1331.
30.	盛伟, 林敏. 中医正骨手法联合康复训练改善颈型颈椎病患者CASCS评分、Berg平衡功能和疼痛程度效果观察. 时珍国医国药, 2018, 29(06): 1391-1393.
31.	冯敏山, 高景华, 高春雨, 等. 坐位腰椎旋转手法的运动学研究. 中国中医骨伤科杂志, 2018, 26(07): 16-20.
32.	曹慧, 高凡茹, 魏德健, 等. 中医正骨仿真系统中姿态检测的应用. 时珍国医国药, 2017, 28(12): 3017-3019.
33.	周晨烁, 魏德健, 曹慧, 等. 中医正骨仿真系统中正骨手法的评价. 中华中医药学刊, 2018, 36(08): 1826-1828.
34.	朱立国, 韩涛, 于杰, 等. 中医骨伤科旋提手法规范化操作传承模式初探. 中医杂志, 2018, 59(11): 927-931.

1. Guerra J, Uddin J, Nilsen D, et al. Capture, learning, and classification of upper extremity movement primitives in healthy controls and stroke patients//2017 International Conference on Rehabilitation Robotics (ICORR).lodon, UK: IEEE, 2017: 547-554.
2. Wang Yufan, Zhao Yuliang, Chan R H, et al. Volleyball skill assessment using a single wearable micro inertial measurement unit at wrist. IEEE Access, 2018, 6: 13758-13765.
3. 潘月浩, 宋执环, 杜往泽, 等. 适用于老年公寓的动作识别设计方法. 电子测量与仪器学报, 2017, 31(1): 29-35.
4. 金刚, 李德华, 周学泳. 表演动画中的运动捕捉技术. 中国图象图形学报, 2000, 5(3): 88-91.
5. 朱红蕾, 朱昶胜, 徐志刚. 人体行为识别数据集研究进展. 自动化学报, 2018, 44(06): 978-1004.
6. Alt Murphy M, Murphy S, Persson H C, et al. Kinematic analysis using 3D motion capture of drinking task in People with and without upper-extremity impairments. J Vis Exp, 2018(133): e57228.
7. Callejas-Cuervo M, Gutierrez R M, Hernandez A I. Joint amplitude MEMS based measurement platform for low cost and high accessibility telerehabilitation: elbow case study. J Bodyw Mov Ther, 2017, 21(3): 574-581.
8. Palumbo F, Gallicchio C, Pucci R, et al. Human activity recognition using multisensor data fusion based on reservoir computing. J Ambient Intell Smart Environ, 2016, 8(2): 87-107.
9. Ryoo M S, Chen C C, Aggarwal J K, et al. An overview of contest on semantic description of human activities (SDHA) 2010//2010 International Conference on Pattern Recognition (ICPP).Berlin, Germany, 2010: 270-285.
10. Wang Jiang, Liu Zicheng, Wu Ying, et al. Mining actionlet ensemble for action recognition with depth cameras//2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).RI, USA: IEEE, 2012: 1290-1297.
11. Chen C, Jafari R, Kehtarnavaz N. UTD-MHAD: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor/2015 IEEE International Conference on Image Processing(ICIP).QC, Canada: IEEE, 2015: 168-172.
12. 王剑, 李春雨, 李跃新. 结合区域HOF和字典学习的人体行为识别方法. 计算机应用研究, 2017, 34(9): 2863-2866.
13. 肖玉玲. 结合HOG/HOF级联特征和多层分类器的人体行为识别. 计算机工程与设计, 2017, 38(9): 2567-2572.
14. Yao Hongxian, Jiang Xinghao, Sun Tanfeng, et al. 3D human action recognition based on the spatial-temporal moving skeleton descriptor//2017 IEEE International Conference on Multimedia and Expo (ICME).Shenzhen, China: IEEE, 2017: 937-942.
15. Shih T Y, Hsieh Y W, WU C Y, et al. Quantifying kinematic features of upper-extremity movement strategies during tooth brushing in patients with stroke and healthy controls. J Neurol Sci, 2017, 381: 995.
16. Hassan M M, Uddin M Z, Mohamed A A. A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems, 2018, 81: 307-313.
17. 胡婷鸿, 万雷, 刘太昂, 等. 深度学习在图像识别及骨龄评估中的优势及应用前景. 法医学杂志, 2017, 33(6): 629-634, 639.
18. Nweke H F, Teh Y W, Al-garadi M A, et al. Deep learning algorithms for human activity recognition using Mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl, 2018, 105: 233-261.
19. 张瑞, 李其申, 储珺. 基于3D卷积神经网络的人体动作识别算法. 计算机工程, 2019, 45(01): 259-263.
20. Pham H H, Khoudour L, Crouzil A, et al. Exploiting deep residual networks for human action recognition from skeletal data. Computer Vision and Image Understanding, 2018, 170:51-66.
21. Li Honggui, Trocan M. Deep learning of smartphone sensor data for personal health assistance. Microelectronics J, 2019, 88: 164-172.
22. 胡双杰, 秦建邦, 郭薇. 基于特征自动提取的跌倒检测算法. 传感技术学报, 2018, 31(12): 1842-1847.
23. 裴晓敏, 范慧杰, 唐延东. 时空特征融合深度学习网络人体行为识别方法. 红外与激光工程, 2018, 47(02): 55-60.
24. Eitel A, Springenberg J T, Spinello L, et al. Multimodal deep learning for robust RGB-D object recognition//2015 IEEE International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany: IEEE, 2015: 681-687.
25. Kim J H, Hong G S, Kim B G, et al. DeepGesture: deep learning-based gesture recognition scheme using motion sensors. Displays, 2018, 55: 38-45.
26. 李玉鹏, 刘婷婷, 张良. 基于深度学习的人体动作识别方法. 计算机应用研究. 2019, 37(1). DOI:10.19734/j.issn.1001-3695.2018.05.0499.
27. 罗会兰, 童康, 孔繁胜. 基于深度学习的视频中人体动作识别进展综述. 电子学报, 2019, 47(05): 1162-1173.
28. Fernandez-Cervantes V, Neubauer N, Hunter B, et al. VirtualGym: A kinect-based system for seniors exercising at home. Entertain Comput, 2018, 27: 60-72.
29. 陈朝晖, 张文娣. 理筋正骨手法结合运动训练治疗慢性颈痛的临床观察. 中国中西医结合杂志, 2018, 38(11): 1327-1331.
30. 盛伟, 林敏. 中医正骨手法联合康复训练改善颈型颈椎病患者CASCS评分、Berg平衡功能和疼痛程度效果观察. 时珍国医国药, 2018, 29(06): 1391-1393.
31. 冯敏山, 高景华, 高春雨, 等. 坐位腰椎旋转手法的运动学研究. 中国中医骨伤科杂志, 2018, 26(07): 16-20.
32. 曹慧, 高凡茹, 魏德健, 等. 中医正骨仿真系统中姿态检测的应用. 时珍国医国药, 2017, 28(12): 3017-3019.
33. 周晨烁, 魏德健, 曹慧, 等. 中医正骨仿真系统中正骨手法的评价. 中华中医药学刊, 2018, 36(08): 1826-1828.
34. 朱立国, 韩涛, 于杰, 等. 中医骨伤科旋提手法规范化操作传承模式初探. 中医杂志, 2018, 59(11): 927-931.

Previous Article
Research advancements of motor imagery for motor function recovery after stroke
Next Article
Research progress of decellularized extracellular matrix hydrogel in regenerative medicine

Journal of Biomedical Engineering

A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content