Robotic arm control system based on augmented reality brain-computer interface and computer vision_Journal of Biomedical Engineering

Authors：

 CHEN Xiaogang ¹ , LI Kun ²

1. Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin 300192, P.R.China;
2. Medical Equipment Management Division of 3201 Hospital, Hanzhong, Shaanxi 723000, P.R.China;

Corresponding author：

CHEN Xiaogang, Email: chenxg@bme.cams.cn

Keywords：

brain-computer interface; robotic arm; steady-state visual evoked potential; augmented reality; computer vision

DOI：

10.7507/1001-5515.202011039

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Brain-computer interface (BCI) has great potential to replace lost upper limb function. Thus, there has been great interest in the development of BCI-controlled robotic arm. However, few studies have attempted to use noninvasive electroencephalography (EEG)-based BCI to achieve high-level control of a robotic arm. In this paper, a high-level control architecture combining augmented reality (AR) BCI and computer vision was designed to control a robotic arm for performing a pick and place task. A steady-state visual evoked potential (SSVEP)-based BCI paradigm was adopted to realize the BCI system. Microsoft's HoloLens was used to build an AR environment and served as the visual stimulator for eliciting SSVEPs. The proposed AR-BCI was used to select the objects that need to be operated by the robotic arm. The computer vision was responsible for providing the location, color and shape information of the objects. According to the outputs of the AR-BCI and computer vision, the robotic arm could autonomously pick the object and place it to specific location. Online results of 11 healthy subjects showed that the average classification accuracy of the proposed system was 91.41%. These results verified the feasibility of combing AR, BCI and computer vision to control a robotic arm, and are expected to provide new ideas for innovative robotic arm control approaches.

Citation： CHEN Xiaogang, LI Kun. Robotic arm control system based on augmented reality brain-computer interface and computer vision. Journal of Biomedical Engineering, 2021, 38(3): 483-491. doi: 10.7507/1001-5515.202011039 Copy

1.	Biasiucci A, Leeb R, Iturrate I, et al. Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nat Commun, 2018, 9(1): 2421.
2.	Pels E G M, Aarnoutse E J, Leinders S, et al. Stability of a chronic implanted brain-computer interface in late-stage amyotrophic lateral sclerosis. Clin Neurophysiol, 2019, 130(10): 1798-1803.
3.	Ganzer P D, Colachis S C 4th, Schwemmer M A, et al. Restoring the sense of touch using a sensorimotor demultiplexing neural interface. Cell, 2020, 181(4): 763-773.
4.	Abdi J, Al-Hindawi A, Ng T, et al. Scoping review on the use of socially assistive robot technology in elderly care. BMJ Open, 2018, 8(2): e018815.
5.	Di Lillo P, Arrichiello F, Di Vito D, et al. BCI-controlled assistive manipulator: developed architecture and experimental results. IEEE Trans Cogn Dev Syst, 2020. DOI: 10.1109/TCDS.2020.2979375.
6.	Abiri R, Borhani S, Sellers E W, et al. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng, 2019, 16(1): 011001.
7.	Si-Mohammed H, Petit J, Jeunet C, et al. Towards BCI-based interfaces for augmented reality: feasibility, design and evaluation. IEEE Trans Vis Comput Graphics, 2020, 26(3): 1608-1621.
8.	McFarland D J. Brain-computer interfaces for amyotrophic lateral sclerosis. Muscle Nerve, 2020, 61(6): 702-707.
9.	McFarland D J, Wolpaw J R. EEG-based brain-computer interfaces. Curr Opin Biomed Eng, 2017, 4: 194-200.
10.	Edelman B J, Meng J, Suma D, et al. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci Robot, 2019, 4(31): eaaw6844.
11.	He Y, Eguren D, Azorín J M, et al. Brain-machine interfaces for controlling lower-limb powered robotic systems. J Neural Eng, 2018, 15(2): 021004.
12.	Chen X, Zhao B, Wang Y, et al. Control of a 7-DOF robotic arm system with an SSVEP-based BCI. Int J Neural Syst, 2018, 28(8): 1850018.
13.	Benabid A L, Costecalde T, Eliseyev A, et al. An exoskeleton controlled by an epidural wireless brain-machine interface in a tetraplegic patient: a proof-of-concept demonstration. Lancet Neurol, 2019, 18(12): 1112-1122.
14.	Chen X, Zhao B, Wang Y, et al. Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. J Neural Eng, 2019, 16(2): 026012.
15.	Ke Y, Liu P, An X, et al. An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng, 2020, 17(1): 016066.
16.	Fernández-Rodríguez Á, Velasco-Álvarez F, Ron-Angevin R. Review of real brain-controlled wheelchairs. J Neural Eng, 2016, 13(6): 061001.
17.	Jeong J H, Shim K H, Kim D J, et al. Brain-controlled robotic arm system based on multi-directional CNN-BiLSTM network using EEG signals. IEEE Trans Neural Syst Rehabil Eng, 2020, 28(5): 1226-1238.
18.	Hortal E, Ianez E, Ubeda A, et al. Combining a brain-machine interface and an electrooculography interface to perform pick and place tasks with a robotic arm. Robot Auton Syst, 2015, 72: 181-188.
19.	Meng J, Zhang S, Bekyo A, et al. Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci Rep, 2016, 6: 38565.
20.	Zeng H, Wang Y, Wu C, et al. Closed-loop hybrid gaze brain-machine interface based robotic arm control with augmented reality feedback. Front Neurorobot, 2017, 11: 60.
21.	Xu Y, Ding C, Shu X, et al. Shared control of a robotic arm using non-invasive brain-computer interface and computer vision guidance. Rob Auton Syst, 2019, 115: 121-129.
22.	Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. J Neural Eng, 2015, 12(4): 046008.
23.	Di Russo F, Spinelli D. Electrophysiological evidence for an early attentional mechanism in visual processing in humans. Vision Res, 1999, 39(18): 2975-2985.
24.	Chen X, Wang Y, Nakanishi M, et al. High-speed spelling with a noninvasive brain-computer interface. Proc Natl Acad Sci USA, 2015, 112(44): E6058-E6067.
25.	Nakanishi M, Wang Y, Chen X, et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans Biomed Eng, 2018, 65(1): 104-112.
26.	Chen X, Wang Y, Zhang S, et al. Effects of stimulation frequency and stimulation waveform on steady-state visual evoked potentials using a computer monitor. J Neural Eng, 2019, 16(6): 066007.
27.	Chen X, Hu N, Wang Y, et al. Validation of a brain-computer interface version of the digit symbol substitution test in healthy subjects. Comput Biol Med, 2020, 120: 103729.
28.	Putze F, Vourvopoulos A, Lécuyer A, et al. Brain-computer interfaces and augmented/virtual reality. Front Hum Neurosci, 2020, 14: 144.
29.	Park S, Cha H S, Im C H. Development of an online home appliance control system using augmented reality and an SSVEP-based brain-computer interface. IEEE Access, 2019, 7: 163604-163614.
30.	Zhao X, Liu C, Xu Z, et al. SSVEP stimulus layout effect on accuracy of brain-computer interfaces in augmented reality glasses. IEEE Access, 2020, 8: 5990-5998.
31.	Chen Y, Yang C, Chen X, et al. A novel training-free recognition method for SSVEP-based BCIs using dynamic window strategy. J Neural Eng, 2020. DOI: 10.1088/1741-2552/ab914e.
32.	Ahn J W, Ku Y, Kim D Y, et al. Wearable in-the-ear EEG system for SSVEP-based brain-computer interface. Electron Lett, 2018, 54(7): 413-414.

1. Biasiucci A, Leeb R, Iturrate I, et al. Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nat Commun, 2018, 9(1): 2421.
2. Pels E G M, Aarnoutse E J, Leinders S, et al. Stability of a chronic implanted brain-computer interface in late-stage amyotrophic lateral sclerosis. Clin Neurophysiol, 2019, 130(10): 1798-1803.
3. Ganzer P D, Colachis S C 4th, Schwemmer M A, et al. Restoring the sense of touch using a sensorimotor demultiplexing neural interface. Cell, 2020, 181(4): 763-773.
4. Abdi J, Al-Hindawi A, Ng T, et al. Scoping review on the use of socially assistive robot technology in elderly care. BMJ Open, 2018, 8(2): e018815.
5. Di Lillo P, Arrichiello F, Di Vito D, et al. BCI-controlled assistive manipulator: developed architecture and experimental results. IEEE Trans Cogn Dev Syst, 2020. DOI: 10.1109/TCDS.2020.2979375.
6. Abiri R, Borhani S, Sellers E W, et al. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng, 2019, 16(1): 011001.
7. Si-Mohammed H, Petit J, Jeunet C, et al. Towards BCI-based interfaces for augmented reality: feasibility, design and evaluation. IEEE Trans Vis Comput Graphics, 2020, 26(3): 1608-1621.
8. McFarland D J. Brain-computer interfaces for amyotrophic lateral sclerosis. Muscle Nerve, 2020, 61(6): 702-707.
9. McFarland D J, Wolpaw J R. EEG-based brain-computer interfaces. Curr Opin Biomed Eng, 2017, 4: 194-200.
10. Edelman B J, Meng J, Suma D, et al. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci Robot, 2019, 4(31): eaaw6844.
11. He Y, Eguren D, Azorín J M, et al. Brain-machine interfaces for controlling lower-limb powered robotic systems. J Neural Eng, 2018, 15(2): 021004.
12. Chen X, Zhao B, Wang Y, et al. Control of a 7-DOF robotic arm system with an SSVEP-based BCI. Int J Neural Syst, 2018, 28(8): 1850018.
13. Benabid A L, Costecalde T, Eliseyev A, et al. An exoskeleton controlled by an epidural wireless brain-machine interface in a tetraplegic patient: a proof-of-concept demonstration. Lancet Neurol, 2019, 18(12): 1112-1122.
14. Chen X, Zhao B, Wang Y, et al. Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. J Neural Eng, 2019, 16(2): 026012.
15. Ke Y, Liu P, An X, et al. An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng, 2020, 17(1): 016066.
16. Fernández-Rodríguez Á, Velasco-Álvarez F, Ron-Angevin R. Review of real brain-controlled wheelchairs. J Neural Eng, 2016, 13(6): 061001.
17. Jeong J H, Shim K H, Kim D J, et al. Brain-controlled robotic arm system based on multi-directional CNN-BiLSTM network using EEG signals. IEEE Trans Neural Syst Rehabil Eng, 2020, 28(5): 1226-1238.
18. Hortal E, Ianez E, Ubeda A, et al. Combining a brain-machine interface and an electrooculography interface to perform pick and place tasks with a robotic arm. Robot Auton Syst, 2015, 72: 181-188.
19. Meng J, Zhang S, Bekyo A, et al. Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci Rep, 2016, 6: 38565.
20. Zeng H, Wang Y, Wu C, et al. Closed-loop hybrid gaze brain-machine interface based robotic arm control with augmented reality feedback. Front Neurorobot, 2017, 11: 60.
21. Xu Y, Ding C, Shu X, et al. Shared control of a robotic arm using non-invasive brain-computer interface and computer vision guidance. Rob Auton Syst, 2019, 115: 121-129.
22. Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. J Neural Eng, 2015, 12(4): 046008.
23. Di Russo F, Spinelli D. Electrophysiological evidence for an early attentional mechanism in visual processing in humans. Vision Res, 1999, 39(18): 2975-2985.
24. Chen X, Wang Y, Nakanishi M, et al. High-speed spelling with a noninvasive brain-computer interface. Proc Natl Acad Sci USA, 2015, 112(44): E6058-E6067.
25. Nakanishi M, Wang Y, Chen X, et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans Biomed Eng, 2018, 65(1): 104-112.
26. Chen X, Wang Y, Zhang S, et al. Effects of stimulation frequency and stimulation waveform on steady-state visual evoked potentials using a computer monitor. J Neural Eng, 2019, 16(6): 066007.
27. Chen X, Hu N, Wang Y, et al. Validation of a brain-computer interface version of the digit symbol substitution test in healthy subjects. Comput Biol Med, 2020, 120: 103729.
28. Putze F, Vourvopoulos A, Lécuyer A, et al. Brain-computer interfaces and augmented/virtual reality. Front Hum Neurosci, 2020, 14: 144.
29. Park S, Cha H S, Im C H. Development of an online home appliance control system using augmented reality and an SSVEP-based brain-computer interface. IEEE Access, 2019, 7: 163604-163614.
30. Zhao X, Liu C, Xu Z, et al. SSVEP stimulus layout effect on accuracy of brain-computer interfaces in augmented reality glasses. IEEE Access, 2020, 8: 5990-5998.
31. Chen Y, Yang C, Chen X, et al. A novel training-free recognition method for SSVEP-based BCIs using dynamic window strategy. J Neural Eng, 2020. DOI: 10.1088/1741-2552/ab914e.
32. Ahn J W, Ku Y, Kim D Y, et al. Wearable in-the-ear EEG system for SSVEP-based brain-computer interface. Electron Lett, 2018, 54(7): 413-414.

Journal of Biomedical Engineering

Robotic arm control system based on augmented reality brain-computer interface and computer vision

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content