A design and evaluation of wearable p300 brain-computer interface system based on Hololens2_Journal of Biomedical Engineering

Authors：

 LI Qi ^1,2 , ZHANG Tingjia ¹ , SONG Yu ¹ , LIU Yulong ² , SUN Meiqi ¹

1. School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, P. R. China;
2. Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, Guangdong 528437, P. R. China;

Corresponding author：

LI Qi, Email: liqi@cust.edu.cn

Keywords：

Brain-computer interface; Augment reality; P300-speller; Wearable

DOI：

10.7507/1001-5515.202207055

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Patients with amyotrophic lateral sclerosis ( ALS ) often have difficulty in expressing their intentions through language and behavior, which prevents them from communicating properly with the outside world and seriously affects their quality of life. The brain-computer interface (BCI) has received much attention as an aid for ALS patients to communicate with the outside world, but the heavy device causes inconvenience to patients in the application process. To improve the portability of the BCI system, this paper proposed a wearable P300-speller brain-computer interface system based on the augmented reality (MR-BCI). This system used Hololens2 augmented reality device to present the paradigm, an OpenBCI device to capture EEG signals, and Jetson Nano embedded computer to process the data. Meanwhile, to optimize the system’s performance for character recognition, this paper proposed a convolutional neural network classification method with low computational complexity applied to the embedded system for real-time classification. The results showed that compared with the P300-speller brain-computer interface system based on the computer screen (CS-BCI), MR-BCI induced an increase in the amplitude of the P300 component, an increase in accuracy of 1.7% and 1.4% in offline and online experiments, respectively, and an increase in the information transfer rate of 0.7 bit/min. The MR-BCI proposed in this paper achieves a wearable BCI system based on guaranteed system performance. It has a positive effect on the realization of the clinical application of BCI.

Citation： LI Qi, ZHANG Tingjia, SONG Yu, LIU Yulong, SUN Meiqi. A design and evaluation of wearable p300 brain-computer interface system based on Hololens2. Journal of Biomedical Engineering, 2023, 40(4): 709-717. doi: 10.7507/1001-5515.202207055 Copy

1.	Collaborators G. A systematic analysis for the global burden of disease study 2017. Lancet, 2018, 392(10159): 1789-1858.
2.	Sellers E W, Donchin E. A P300-based brain-computer interface: initial tests by ALS patients. Clin Neurophysiol, 2006, 117(3): 538-548.
3.	Mcfarland D J, Wolpaw J R. EEG-based brain-computer interfaces. Curr Opin Biomed Eng, 2017, 4: 194-200.
4.	Abootalebi V, Moradi M H, Khalilzadeh M A. A comparison of methods for ERP assessment in a P300-based GKT. Int J Psychophysiol, 2006, 62(2): 309-320.
5.	Dien J, Spencer K M, Donchin E J P. Parsing the late positive complex: mental chronometry and the ERP components that inhabit the neighborhood of the P300. Psychophysiology, 2004, 41(5): 665-678.
6.	Schembri P, Pelc M, Ma J. The effect that auditory distractions have on a visual P300 speller while utilizing low-cost off-the-shelf equipment. Computers, 2020, 9(3): 68.
7.	Qin Z, Li Q. High rate BCI with portable devices based on EEG. Smart Health, 2018, 9: 115-128.
8.	Kerous B, Liarokapis F. BrainChat—A collaborative augmented reality brain interface for message communication// Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). Nantes: IEEE, 2017: 279-283.
9.	Ke Y F, Liu P X, An X W, et al. An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng, 2020, 17(1): 160-166.
10.	Zhu D, Zhou X, Guo Q. An identification system based on portable EEG acquisition equipment// Proceedings of the 2013 Third International Conference on Intelligent System Design and Engineering Applications. Hong Kong: IEEE, 2013: 281-284.
11.	Chen S C, Hsieh S C, Liang C K. An intelligent brain computer interface of visual evoked potential EEG// 2008 Eighth International Conference on Intelligent Systems Design and Applications. Kaohsuing: IEEE, 2008: 343-346.
12.	Frey J. Comparison of a consumer grade EEG amplifier with medical grade equipment in BCI applications// Proceedings of the International BCI meeting. New York: IEEE, 2016: hal-01278245.
13.	Lu Z H, Li Q, Gao N, et al. Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller. Brain Behav, 2019, 9(12): e01479.
14.	Ramirez-Quintana J A, Madrid-Herrera L, Chacon-Murguia M I, et al. Brain-computer interface system based on p300 processing with convolutional neural network, novel speller, and low number of electrodes. Cogn Comput, 2021, 13(1): 108-124.
15.	Thompson D E, Blain-Moraes S, Huggins J E. Performance assessment in brain-computer interface-based augmentative and alternative communication. BioMed Eng OnLine, 2013, 12(1): 43.
16.	Gao F, Cao B, Cao Y, et al. Electrophysiological evidence of separate pathways for the perception of depth and 3D objects. Int J Psychophysiol, 2015, 96(2): 65-73.
17.	Cauquil A S, Trotter Y, Taylor M J. At what stage of neural processing do perspective depth cues make a difference?. Exp Brain Res, 2006, 170(4): 457-463.
18.	Cattan G, Andreev A, Mendoza C, et al. A comparison of mobile VR display running on an ordinary smartphone with standard PC display for P300-BCI stimulus presentation. IEEE T Games, 2019, 13(1): 68-77.
19.	Lenhardt A, Ritter H. An augmented-reality based brain-computer interface for robot control// Proceedings of the Neural Information Processing Models and Applications—17th International Conference (ICONIP). Sydney: APNNS, 2010: 58-65.
20.	Kouji T, Naoki H, Kenji K. Towards intelligent environments: An augmented reality–brain–machine interface operated with a see-through head-mount display. Front Neurosci-Switz, 2011, 5: 60.
21.	Uno K, Naito G, Tobisa Y, et al. Basic investigation of brain–computer interface combined with augmented reality and development of an improvement method using the nontarget object. Electr Commun Jap, 2015, 98(8): 9-15.
22.	Rohani D A, Puthusserypdy S. BCI inside a virtual reality classroom: a potential training tool for attention. EPJ NBP, 2015, 3(1): 1-14.
23.	Zhong S, Liu Y, Yu Y, et al. A dynamic user interface based BCI environmental control system. Int J Hum-Comput Int, 2020, 36(1): 55-66.

1. Collaborators G. A systematic analysis for the global burden of disease study 2017. Lancet, 2018, 392(10159): 1789-1858.
2. Sellers E W, Donchin E. A P300-based brain-computer interface: initial tests by ALS patients. Clin Neurophysiol, 2006, 117(3): 538-548.
3. Mcfarland D J, Wolpaw J R. EEG-based brain-computer interfaces. Curr Opin Biomed Eng, 2017, 4: 194-200.
4. Abootalebi V, Moradi M H, Khalilzadeh M A. A comparison of methods for ERP assessment in a P300-based GKT. Int J Psychophysiol, 2006, 62(2): 309-320.
5. Dien J, Spencer K M, Donchin E J P. Parsing the late positive complex: mental chronometry and the ERP components that inhabit the neighborhood of the P300. Psychophysiology, 2004, 41(5): 665-678.
6. Schembri P, Pelc M, Ma J. The effect that auditory distractions have on a visual P300 speller while utilizing low-cost off-the-shelf equipment. Computers, 2020, 9(3): 68.
7. Qin Z, Li Q. High rate BCI with portable devices based on EEG. Smart Health, 2018, 9: 115-128.
8. Kerous B, Liarokapis F. BrainChat—A collaborative augmented reality brain interface for message communication// Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). Nantes: IEEE, 2017: 279-283.
9. Ke Y F, Liu P X, An X W, et al. An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng, 2020, 17(1): 160-166.
10. Zhu D, Zhou X, Guo Q. An identification system based on portable EEG acquisition equipment// Proceedings of the 2013 Third International Conference on Intelligent System Design and Engineering Applications. Hong Kong: IEEE, 2013: 281-284.
11. Chen S C, Hsieh S C, Liang C K. An intelligent brain computer interface of visual evoked potential EEG// 2008 Eighth International Conference on Intelligent Systems Design and Applications. Kaohsuing: IEEE, 2008: 343-346.
12. Frey J. Comparison of a consumer grade EEG amplifier with medical grade equipment in BCI applications// Proceedings of the International BCI meeting. New York: IEEE, 2016: hal-01278245.
13. Lu Z H, Li Q, Gao N, et al. Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller. Brain Behav, 2019, 9(12): e01479.
14. Ramirez-Quintana J A, Madrid-Herrera L, Chacon-Murguia M I, et al. Brain-computer interface system based on p300 processing with convolutional neural network, novel speller, and low number of electrodes. Cogn Comput, 2021, 13(1): 108-124.
15. Thompson D E, Blain-Moraes S, Huggins J E. Performance assessment in brain-computer interface-based augmentative and alternative communication. BioMed Eng OnLine, 2013, 12(1): 43.
16. Gao F, Cao B, Cao Y, et al. Electrophysiological evidence of separate pathways for the perception of depth and 3D objects. Int J Psychophysiol, 2015, 96(2): 65-73.
17. Cauquil A S, Trotter Y, Taylor M J. At what stage of neural processing do perspective depth cues make a difference?. Exp Brain Res, 2006, 170(4): 457-463.
18. Cattan G, Andreev A, Mendoza C, et al. A comparison of mobile VR display running on an ordinary smartphone with standard PC display for P300-BCI stimulus presentation. IEEE T Games, 2019, 13(1): 68-77.
19. Lenhardt A, Ritter H. An augmented-reality based brain-computer interface for robot control// Proceedings of the Neural Information Processing Models and Applications—17th International Conference (ICONIP). Sydney: APNNS, 2010: 58-65.
20. Kouji T, Naoki H, Kenji K. Towards intelligent environments: An augmented reality–brain–machine interface operated with a see-through head-mount display. Front Neurosci-Switz, 2011, 5: 60.
21. Uno K, Naito G, Tobisa Y, et al. Basic investigation of brain–computer interface combined with augmented reality and development of an improvement method using the nontarget object. Electr Commun Jap, 2015, 98(8): 9-15.
22. Rohani D A, Puthusserypdy S. BCI inside a virtual reality classroom: a potential training tool for attention. EPJ NBP, 2015, 3(1): 1-14.
23. Zhong S, Liu Y, Yu Y, et al. A dynamic user interface based BCI environmental control system. Int J Hum-Comput Int, 2020, 36(1): 55-66.

Journal of Biomedical Engineering

A design and evaluation of wearable p300 brain-computer interface system based on Hololens2

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content