The supernumerary robotic limbs of brain-computer interface based on asynchronous steady-state visual evoked potential_Journal of Biomedical Engineering

Authors：

XIE Ping ^1,2 , MEN Yandi ^1,2 , ZHEN Jiale ^1,2 , SHAO Xiening ^1,2 , ZHAO Jing ¹ ,  CHEN Xiaoling ^1,2

1. College of Electrical Engineering, Yanshan University, Qinhuangdao, Hebei 066000, P. R. China;
2. Key Lab of Intelligent Rehabilitation and Neuromodulation of Hebei Province, Qinhuangdao, Hebei 066000, P. R. China;

Corresponding author：

CHEN Xiaoling, Email: xlchen@ysu.edu.cn

Keywords：

Asynchronous brain-computer interface; Steady-state visual evoked potential; Supernumerary robotic limb; Augmented reality

DOI：

10.7507/1001-5515.202312056

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Brain-computer interface (BCI) based on steady-state visual evoked potential (SSVEP) have attracted much attention in the field of intelligent robotics. Traditional SSVEP-based BCI systems mostly use synchronized triggers without identifying whether the user is in the control or non-control state, resulting in a system that lacks autonomous control capability. Therefore, this paper proposed a SSVEP asynchronous state recognition method, which constructs an asynchronous state recognition model by fusing multiple time-frequency domain features of electroencephalographic (EEG) signals and combining with a linear discriminant analysis (LDA) to improve the accuracy of SSVEP asynchronous state recognition. Furthermore, addressing the control needs of disabled individuals in multitasking scenarios, a brain-machine fusion system based on SSVEP-BCI asynchronous cooperative control was developed. This system enabled the collaborative control of wearable manipulator and robotic arm, where the robotic arm acts as a “third hand”, offering significant advantages in complex environments. The experimental results showed that using the SSVEP asynchronous control algorithm and brain-computer fusion system proposed in this paper could assist users to complete multitasking cooperative operations. The average accuracy of user intent recognition in online control experiments was 93.0%, which provides a theoretical and practical basis for the practical application of the asynchronous SSVEP-BCI system.

Citation： XIE Ping, MEN Yandi, ZHEN Jiale, SHAO Xiening, ZHAO Jing, CHEN Xiaoling. The supernumerary robotic limbs of brain-computer interface based on asynchronous steady-state visual evoked potential. Journal of Biomedical Engineering, 2024, 41(4): 664-672. doi: 10.7507/1001-5515.202312056 Copy

1.	Xu Dongcen, Tang Fengzhen, Li Yiping, et al. An analysis of deep learning models in SSVEP-based BCI: A survey. Brain Sci, 2023, 13(3): 483-483.
2.	李奇, 张庭嘉, 宋雨, 等. 基于Hololens2的可穿戴式P300脑机接口系统设计与评估. 生物医学工程学杂志, 2023, 40(4): 709-717.
3.	Lazcano-Herrera A G, Fuentes-Aguilar R Q, Chairez I, et al. Review on BCI virtual rehabilitation and remote technology based on EEG for assistive devices. Appl Sci, 2022, 12(23): 12253.
4.	Gueye T, Dedkova M, Rogalewicz V, et al. Early post-stroke rehabilitation for upper limb motor function using virtual reality and exoskeleton: equally efficient in older patients. Neurol Neurochir Pol, 2020, 55(1): 91-96.
5.	Aljalal M, Ibrahim S, Djemal R, et al. Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals. Intel Serv Robot, 2020, 13: 539-563.
6.	Penaloza I C, Nishio S. BMI control of a third arm for multitasking. Sci Robot, 2018, 3(20): eaat1228.
7.	Paulina K, Danielle C, O R M, et al. Robotic hand augmentation drives changes in neural body representation. Sci Robot, 2021, 6(54). DOI: 10.1126/scirobotics.abd7935.
8.	赵思恺, 李长乐, 张宗伟, 等. 模块化可重构外肢体机器人. 仪器仪表学报, 2021, 42(4): 218-227.
9.	Tong Y, Liu J. Review of research and development of supernumerary robotic limbs. IEEE/CAA J Autom Sinica, 2021, 8(5): 929-952.
10.	Baldi T L, D’Aurizio N, Gurgone S, et al. Exploiting intrinsic kinematic null space for supernumerary robotic limbs control// 2023 IEEE International Conference on Robotics and Automation (ICRA). London: IEEE, 2023: 11957-11963.
11.	Tu Z, Fang Y, Leng Y, et al. Task-based human-robot collaboration control of supernumerary robotic limbs for overhead tasks. IEEE Robot Autom Lett, 2023, 8(8): 4505-4512.
12.	Guggenheim J, Hoffman R, Song H, et al. Leveraging the human operator in the design and control of supernumerary robotic limbs. IEEE Robot Autom Lett, 2020, 5(2): 2177-2184.
13.	Eden J, Bräcklein M, Ibáñez J, et al. Principles of human movement augmentation and the challenges in making it a reality. Nat Commun, 2022, 13(1): 1345.
14.	Yang B, Huang J, Chen X, et al. Supernumerary robotic limbs: a review and future outlook. IEEE Trans Med Robot Bionics, 2021, 3(3): 623-639.
15.	Zhang Y, Xie S Q, Wang H, et al. Bayesian-based classification confidence estimation for enhancing SSVEP detection. IEEE T Instrum Meas, 2023, 72: 1-12.
16.	Hong J, Qin X. Signal processing algorithms for SSVEP-based brain computer interface: State-of-the-art and recent developments. J Intell Fuzzy Syst, 2021, 40(6): 10559-10573.
17.	Ojha M K, Mukul M K. Detection of target frequency from SSVEP signal using empirical mode decomposition for SSVEP based BCI inference system. Wireless Pers Commun, 2021, 116: 777-789.
18.	Liu P, Ke Y, Du J, et al. An SSVEP-BCI in augmented reality// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Berlin: IEEE, 2019: 5548-5551.
19.	Zhang S, Gao X, Chen X. Humanoid robot walking in maze controlled by SSVEP-BCI based on augmented reality stimulus. Front Hum Neurosci, 2022, 16: 908050.
20.	Arpaia P, De Benedetto E, Duraccio L. Design, implementation, and metrological characterization of a wearable, integrated AR-BCI hands-free system for health 4. 0 monitoring. Measurement, 2021, 177: 109280.
21.	Zhou Y, He S, Huang Q, et al. A hybrid asynchronous brain-computer interface combining SSVEP and EOG signals. IEEE Trans Biomed, 2020, 67(10): 2881-2892.
22.	Xie J, Zhang H, Liu Y, et al. Asynchronous steady-state visual evoked potential brain-computer interface application: True and false positive rate comparison between with and without eye-tracking switch paradigms// 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR). Xining: IEEE, 2021: 438-443.
23.	Nagel S, Spüler M. Asynchronous non-invasive high-speed BCI speller with robust non-control state detection. Sci Rep, 2019, 9(1): 8269.
24.	Brainard D H, Vision S. The psychophysics toolbox. Spat Vis, 1997, 10(4): 433-436.
25.	陈小刚. 高速率稳态视觉诱发电位脑-机接口的关键技术研究. 北京: 清华大学, 2015.
26.	Ke Y, Du J, Liu S, et al. Enhancing detection of control state for high-speed asynchronous SSVEP-BCIs using frequency-specific framework. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 1405-1417.
27.	Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface. J Neural Eng, 2015, 12(4): 046008.
28.	Zhang R, Cao L, Xu Z, et al. Improving AR-SSVEP recognition accuracy under high ambient brightness through iterative learning. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 1796-1806.
29.	Zhang R, Xu Z, Zhang L, et al. The effect of stimulus number on the recognition accuracy and information transfer rate of SSVEP–BCI in augmented reality. J Neural Eng, 2022, 19(3): 036010.
30.	Liu B, Huang X, Wang Y, et al. BETA: A large benchmark database toward SSVEP-BCI application. Front Neurosci, 2020, 14: 627.
31.	Lin Z, Zhang C, Wu W, et al. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng, 2006, 53(12): 2610-2614.
32.	Sözer A T, Bülent C. Novel detection features for SSVEP based BCI: Coefficient of variation and variation speed. Brain, 2017, 8(2): 144-150.
33.	Tharwat A, Gaber T, Ibrahim A, et al. Linear discriminant analysis: A detailed tutorial. AI Commun, 2017, 30(2): 169-190.
34.	Wang S, Ji B, Shao D, et al. A methodology for enhancing SSVEP features using adaptive filtering based on the spatial distribution of EEG signals. Micromachines, 2023, 14(5): 976.
35.	Xu Z, Chen G, Zhang R. Boosters of the metaverse: a review of augmented reality-based brain-computer interface. Bacomics, 2024, 3(1): 2305962.

1. Xu Dongcen, Tang Fengzhen, Li Yiping, et al. An analysis of deep learning models in SSVEP-based BCI: A survey. Brain Sci, 2023, 13(3): 483-483.
2. 李奇, 张庭嘉, 宋雨, 等. 基于Hololens2的可穿戴式P300脑机接口系统设计与评估. 生物医学工程学杂志, 2023, 40(4): 709-717.
3. Lazcano-Herrera A G, Fuentes-Aguilar R Q, Chairez I, et al. Review on BCI virtual rehabilitation and remote technology based on EEG for assistive devices. Appl Sci, 2022, 12(23): 12253.
4. Gueye T, Dedkova M, Rogalewicz V, et al. Early post-stroke rehabilitation for upper limb motor function using virtual reality and exoskeleton: equally efficient in older patients. Neurol Neurochir Pol, 2020, 55(1): 91-96.
5. Aljalal M, Ibrahim S, Djemal R, et al. Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals. Intel Serv Robot, 2020, 13: 539-563.
6. Penaloza I C, Nishio S. BMI control of a third arm for multitasking. Sci Robot, 2018, 3(20): eaat1228.
7. Paulina K, Danielle C, O R M, et al. Robotic hand augmentation drives changes in neural body representation. Sci Robot, 2021, 6(54). DOI: 10.1126/scirobotics.abd7935.
8. 赵思恺, 李长乐, 张宗伟, 等. 模块化可重构外肢体机器人. 仪器仪表学报, 2021, 42(4): 218-227.
9. Tong Y, Liu J. Review of research and development of supernumerary robotic limbs. IEEE/CAA J Autom Sinica, 2021, 8(5): 929-952.
10. Baldi T L, D’Aurizio N, Gurgone S, et al. Exploiting intrinsic kinematic null space for supernumerary robotic limbs control// 2023 IEEE International Conference on Robotics and Automation (ICRA). London: IEEE, 2023: 11957-11963.
11. Tu Z, Fang Y, Leng Y, et al. Task-based human-robot collaboration control of supernumerary robotic limbs for overhead tasks. IEEE Robot Autom Lett, 2023, 8(8): 4505-4512.
12. Guggenheim J, Hoffman R, Song H, et al. Leveraging the human operator in the design and control of supernumerary robotic limbs. IEEE Robot Autom Lett, 2020, 5(2): 2177-2184.
13. Eden J, Bräcklein M, Ibáñez J, et al. Principles of human movement augmentation and the challenges in making it a reality. Nat Commun, 2022, 13(1): 1345.
14. Yang B, Huang J, Chen X, et al. Supernumerary robotic limbs: a review and future outlook. IEEE Trans Med Robot Bionics, 2021, 3(3): 623-639.
15. Zhang Y, Xie S Q, Wang H, et al. Bayesian-based classification confidence estimation for enhancing SSVEP detection. IEEE T Instrum Meas, 2023, 72: 1-12.
16. Hong J, Qin X. Signal processing algorithms for SSVEP-based brain computer interface: State-of-the-art and recent developments. J Intell Fuzzy Syst, 2021, 40(6): 10559-10573.
17. Ojha M K, Mukul M K. Detection of target frequency from SSVEP signal using empirical mode decomposition for SSVEP based BCI inference system. Wireless Pers Commun, 2021, 116: 777-789.
18. Liu P, Ke Y, Du J, et al. An SSVEP-BCI in augmented reality// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Berlin: IEEE, 2019: 5548-5551.
19. Zhang S, Gao X, Chen X. Humanoid robot walking in maze controlled by SSVEP-BCI based on augmented reality stimulus. Front Hum Neurosci, 2022, 16: 908050.
20. Arpaia P, De Benedetto E, Duraccio L. Design, implementation, and metrological characterization of a wearable, integrated AR-BCI hands-free system for health 4. 0 monitoring. Measurement, 2021, 177: 109280.
21. Zhou Y, He S, Huang Q, et al. A hybrid asynchronous brain-computer interface combining SSVEP and EOG signals. IEEE Trans Biomed, 2020, 67(10): 2881-2892.
22. Xie J, Zhang H, Liu Y, et al. Asynchronous steady-state visual evoked potential brain-computer interface application: True and false positive rate comparison between with and without eye-tracking switch paradigms// 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR). Xining: IEEE, 2021: 438-443.
23. Nagel S, Spüler M. Asynchronous non-invasive high-speed BCI speller with robust non-control state detection. Sci Rep, 2019, 9(1): 8269.
24. Brainard D H, Vision S. The psychophysics toolbox. Spat Vis, 1997, 10(4): 433-436.
25. 陈小刚. 高速率稳态视觉诱发电位脑-机接口的关键技术研究. 北京: 清华大学, 2015.
26. Ke Y, Du J, Liu S, et al. Enhancing detection of control state for high-speed asynchronous SSVEP-BCIs using frequency-specific framework. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 1405-1417.
27. Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface. J Neural Eng, 2015, 12(4): 046008.
28. Zhang R, Cao L, Xu Z, et al. Improving AR-SSVEP recognition accuracy under high ambient brightness through iterative learning. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 1796-1806.
29. Zhang R, Xu Z, Zhang L, et al. The effect of stimulus number on the recognition accuracy and information transfer rate of SSVEP–BCI in augmented reality. J Neural Eng, 2022, 19(3): 036010.
30. Liu B, Huang X, Wang Y, et al. BETA: A large benchmark database toward SSVEP-BCI application. Front Neurosci, 2020, 14: 627.
31. Lin Z, Zhang C, Wu W, et al. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng, 2006, 53(12): 2610-2614.
32. Sözer A T, Bülent C. Novel detection features for SSVEP based BCI: Coefficient of variation and variation speed. Brain, 2017, 8(2): 144-150.
33. Tharwat A, Gaber T, Ibrahim A, et al. Linear discriminant analysis: A detailed tutorial. AI Commun, 2017, 30(2): 169-190.
34. Wang S, Ji B, Shao D, et al. A methodology for enhancing SSVEP features using adaptive filtering based on the spatial distribution of EEG signals. Micromachines, 2023, 14(5): 976.
35. Xu Z, Chen G, Zhang R. Boosters of the metaverse: a review of augmented reality-based brain-computer interface. Bacomics, 2024, 3(1): 2305962.

Journal of Biomedical Engineering

The supernumerary robotic limbs of brain-computer interface based on asynchronous steady-state visual evoked potential

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content