Multi-task motor imagery electroencephalogram classification based on adaptive time-frequency common spatial pattern combined with convolutional neural network_Journal of Biomedical Engineering

Authors：

HU Ying ^1,2 , LIU Yan ^2,6 , CHENG Chenchen ³ , GENG Chen ² , DAI Bin ^1,2 , PENG Bo ² , ZHU Jianbing ⁴ ,  DAI Yakang ^2,5,6

1. School of Biomedical Engineering, University of Science and Technology of China, Hefei 230026, P. R. China;
2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China;
3. School of Automation, Harbin University of Science and Technology, Harbin 150080, P. R. China;
4. Suzhou Science & Technology Town Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu 215163, P. R. China;
5. Suzhou Guoke Medical Technology Development (Group) Co., LTD, Suzhou, Jiangsu 215163, P. R. China;
6. Suzhou Guoke Kangcheng Medical Technology Co., LTD, Suzhou, Jiangsu 215163, P. R. China;

Corresponding author：

DAI Yakang, Email: daiyk@sibet.ac.cn

Keywords：

Brain-computer interface; Motor imagery electroencephalogram; Common spatial pattern; Spatiotemporal-frequency domain feature fusion; Convolutional neural network

DOI：

10.7507/1001-5515.202206052

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The effective classification of multi-task motor imagery electroencephalogram (EEG) is helpful to achieve accurate multi-dimensional human-computer interaction, and the high frequency domain specificity between subjects can improve the classification accuracy and robustness. Therefore, this paper proposed a multi-task EEG signal classification method based on adaptive time-frequency common spatial pattern (CSP) combined with convolutional neural network (CNN). The characteristics of subjects' personalized rhythm were extracted by adaptive spectrum awareness, and the spatial characteristics were calculated by using the one-versus-rest CSP, and then the composite time-domain characteristics were characterized to construct the spatial-temporal frequency multi-level fusion features. Finally, the CNN was used to perform high-precision and high-robust four-task classification. The algorithm in this paper was verified by the self-test dataset containing 10 subjects (33 ± 3 years old, inexperienced) and the dataset of the 4th 2018 Brain-Computer Interface Competition (BCI competition Ⅳ-2a). The average accuracy of the proposed algorithm for the four-task classification reached 93.96% and 84.04%, respectively. Compared with other advanced algorithms, the average classification accuracy of the proposed algorithm was significantly improved, and the accuracy range error between subjects was significantly reduced in the public dataset. The results show that the proposed algorithm has good performance in multi-task classification, and can effectively improve the classification accuracy and robustness.

Citation： HU Ying, LIU Yan, CHENG Chenchen, GENG Chen, DAI Bin, PENG Bo, ZHU Jianbing, DAI Yakang. Multi-task motor imagery electroencephalogram classification based on adaptive time-frequency common spatial pattern combined with convolutional neural network. Journal of Biomedical Engineering, 2022, 39(6): 1065-1073, 1081. doi: 10.7507/1001-5515.202206052 Copy

1.	Vasiljevic G A M, Miranda L C. Brain–computer interface games based on consumer-grade EEG devices: a systematic literature review. Int J Hum-Comput Int, 2020, 36(2): 105-142.
2.	Chaudhary U, Birbaumer N, Ramos-Murguialday A. Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol, 2016, 12 (9): 513-525.
3.	Abiri R, Borhani S, Sellers E W, et al. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng, 2019, 16 (1): 011001.
4.	Lotte F, Congedo M, Lécuyer A, et al. A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng, 2007, 4(2): R1-R13.
5.	刘锦, 吴小培, 周蚌艳, 等. 单次样本对的CSP滤波器设计及其在脑电训练样本优化中的应用. 信号处理, 2017, 33(7): 993-1001.
6.	Ramoser H, Muller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng, 2000, 8(4): 441-446.
7.	Lotte F, Guan C. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Trans Biomed Eng, 2011, 58(2): 355-362.
8.	Song X, Yoon S C. Improving brain-computer interface classification using adaptive common spatial patterns. Comput Biol Med, 2015, 61: 150-160.
9.	Jin J, Miao Y, Daly I, et al. Correlation based channel selection and regularized feature optimization for MI based BCI. Neural Netw, 2019, 118: 262-270.
10.	于沐涵, 陈峰. 基于HCSP和模糊熵的脑电信号分类. 计算机工程与设计, 2018, 39(2): 557-562.
11.	Ang K K, Chin Z Y, Zhang H H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface// 2008 IEEE International Joint Conference on Neural Networks (IJCNN), Hong Kong: IEEE, 2008: 2390-2397.
12.	Das R, Lopez P S, Khan M A, et al. FBCSP and adaptive boosting for multiclass motor imagery BCI data classification: a machine learning approach//2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto: IEEE, 2020: 1275-1279.
13.	Zhang S, Zhu Z, Zhang B, et al. The CSP-based new features plus non-convex log sparse feature selection for motor imagery EEG classification. Sensors (Basel). 2020, 20(17): 4749.
14.	Blanco-Diaz C F, Antelis J M, Ruiz-Olaya A F. Comparative analysis of spectral and temporal combinations in CSP-based methods for decoding hand motor imagery tasks. J Neurosci Methods. 2022, 371: 109495.
15.	Pei Y, Luo Z G, Zhao H Y, et al. A tensor-based frequency features combination method for brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng. 2022, 30: 465-475.
16.	Altuwaijri G A, Muhammad G. A multibranch of convolutional neural network models for electroencephalogram-based motor imagery classification. Biosensors, 2022, 12(1): 22.
17.	Kumar S, Sharma R, Sharma A. OPTICAL+: a frequency-based deep learning scheme for recognizing brain wave signals. Peerj Comput Sci, 2021, 7: e375.
18.	Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng, 2018, 15(3): 031005.
19.	Hou Y, Chen T, Lun X, et al. A novel method for classification of multi-class motor imagery tasks based on feature fusion. Neurosci Res, 2022, 176: 40-48.
20.	Rashid M, Sulaiman N, Abdul Majeed A P P, et al. Current status, challenges, and possible solutions of EEG-based brain-computer interface: a comprehensive review. Front Neurorobotics, 2020, 14: 25.
21.	Zhang R, Zong Q, Dou L, et al. A novel hybrid deep learning scheme for four-class motor imagery classification. J Neural Eng, 2019, 16(6): 066004.
22.	Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
23.	Amin S U, Alsulaiman M, Muhammad G, et al. Deep learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Generation Computer Systems, 2019, 101: 542–554.
24.	Lun X, Yu Z, Chen T, et al. A simplified CNN classification method for MI-EEG via the electrode pairs signals. Front Hum neurosci, 2020, 14: 338.
25.	Xu S, Zhu L, Kong W, et al. A novel classification method for EEG-based motor imagery with narrow band spatial filters and deep convolutional neural network. Cogn Neurodyn, 2022, 16(2): 379-389.
26.	Ma X, Wang D, Liu D, et al. DWT and CNN based multi-class motor imagery electroencephalographic signal recognition. J Neural Eng, 2020, 17 (1): 016073.
27.	乔晓艳, 王春晖, 任兆麟. 小波统计方法提取想象运动诱发脑电特征. 测试技术学报, 2013, 27(3): 201-207.
28.	张义, 潘赛虎, 邹凌, 等. 基于OVR-CSP的情绪认知重评脑电信号晚正成分研究. 生物医学工程学杂志, 2014, 31(6): 1212-1217.
29.	曾庆山, 范明莉, 宋庆祥. 基于CSP与卷积神经网络算法的多类运动想象脑电信号分类. 科学技术与工程, 2017, 17(27): 144-149.
30.	Stieger J R, Engel S A, Suma D, et al. Benefits of deep learning classification of continuous noninvasive brain-computer interface control. J Neural Eng, 2021,18(4): 046082.
31.	Ang K K, Chin Z Y, Wang C, et al. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci, 2012, 6: 39.
32.	Luo T J, Zhou C L, Chao F. Exploring spatial-frequency-sequential relationships for motor imagery classification with recurrent neural network. BMC Bioinformatics, 2018, 19(1): 344-362.
33.	Liu C, Jing J, Daly I, et al. SincNet-based hybrid neural network for motor imagery EEG decoding. IEEE Trans Neural Syst Rehabil Eng, 2020, 30: 540-549.
34.	Roy A M. Adaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface. Eng Appl Artif Intel, 2022, 116: 105347.

1. Vasiljevic G A M, Miranda L C. Brain–computer interface games based on consumer-grade EEG devices: a systematic literature review. Int J Hum-Comput Int, 2020, 36(2): 105-142.
2. Chaudhary U, Birbaumer N, Ramos-Murguialday A. Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol, 2016, 12 (9): 513-525.
3. Abiri R, Borhani S, Sellers E W, et al. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng, 2019, 16 (1): 011001.
4. Lotte F, Congedo M, Lécuyer A, et al. A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng, 2007, 4(2): R1-R13.
5. 刘锦, 吴小培, 周蚌艳, 等. 单次样本对的CSP滤波器设计及其在脑电训练样本优化中的应用. 信号处理, 2017, 33(7): 993-1001.
6. Ramoser H, Muller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng, 2000, 8(4): 441-446.
7. Lotte F, Guan C. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Trans Biomed Eng, 2011, 58(2): 355-362.
8. Song X, Yoon S C. Improving brain-computer interface classification using adaptive common spatial patterns. Comput Biol Med, 2015, 61: 150-160.
9. Jin J, Miao Y, Daly I, et al. Correlation based channel selection and regularized feature optimization for MI based BCI. Neural Netw, 2019, 118: 262-270.
10. 于沐涵, 陈峰. 基于HCSP和模糊熵的脑电信号分类. 计算机工程与设计, 2018, 39(2): 557-562.
11. Ang K K, Chin Z Y, Zhang H H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface// 2008 IEEE International Joint Conference on Neural Networks (IJCNN), Hong Kong: IEEE, 2008: 2390-2397.
12. Das R, Lopez P S, Khan M A, et al. FBCSP and adaptive boosting for multiclass motor imagery BCI data classification: a machine learning approach//2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto: IEEE, 2020: 1275-1279.
13. Zhang S, Zhu Z, Zhang B, et al. The CSP-based new features plus non-convex log sparse feature selection for motor imagery EEG classification. Sensors (Basel). 2020, 20(17): 4749.
14. Blanco-Diaz C F, Antelis J M, Ruiz-Olaya A F. Comparative analysis of spectral and temporal combinations in CSP-based methods for decoding hand motor imagery tasks. J Neurosci Methods. 2022, 371: 109495.
15. Pei Y, Luo Z G, Zhao H Y, et al. A tensor-based frequency features combination method for brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng. 2022, 30: 465-475.
16. Altuwaijri G A, Muhammad G. A multibranch of convolutional neural network models for electroencephalogram-based motor imagery classification. Biosensors, 2022, 12(1): 22.
17. Kumar S, Sharma R, Sharma A. OPTICAL+: a frequency-based deep learning scheme for recognizing brain wave signals. Peerj Comput Sci, 2021, 7: e375.
18. Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng, 2018, 15(3): 031005.
19. Hou Y, Chen T, Lun X, et al. A novel method for classification of multi-class motor imagery tasks based on feature fusion. Neurosci Res, 2022, 176: 40-48.
20. Rashid M, Sulaiman N, Abdul Majeed A P P, et al. Current status, challenges, and possible solutions of EEG-based brain-computer interface: a comprehensive review. Front Neurorobotics, 2020, 14: 25.
21. Zhang R, Zong Q, Dou L, et al. A novel hybrid deep learning scheme for four-class motor imagery classification. J Neural Eng, 2019, 16(6): 066004.
22. Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
23. Amin S U, Alsulaiman M, Muhammad G, et al. Deep learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Generation Computer Systems, 2019, 101: 542–554.
24. Lun X, Yu Z, Chen T, et al. A simplified CNN classification method for MI-EEG via the electrode pairs signals. Front Hum neurosci, 2020, 14: 338.
25. Xu S, Zhu L, Kong W, et al. A novel classification method for EEG-based motor imagery with narrow band spatial filters and deep convolutional neural network. Cogn Neurodyn, 2022, 16(2): 379-389.
26. Ma X, Wang D, Liu D, et al. DWT and CNN based multi-class motor imagery electroencephalographic signal recognition. J Neural Eng, 2020, 17 (1): 016073.
27. 乔晓艳, 王春晖, 任兆麟. 小波统计方法提取想象运动诱发脑电特征. 测试技术学报, 2013, 27(3): 201-207.
28. 张义, 潘赛虎, 邹凌, 等. 基于OVR-CSP的情绪认知重评脑电信号晚正成分研究. 生物医学工程学杂志, 2014, 31(6): 1212-1217.
29. 曾庆山, 范明莉, 宋庆祥. 基于CSP与卷积神经网络算法的多类运动想象脑电信号分类. 科学技术与工程, 2017, 17(27): 144-149.
30. Stieger J R, Engel S A, Suma D, et al. Benefits of deep learning classification of continuous noninvasive brain-computer interface control. J Neural Eng, 2021,18(4): 046082.
31. Ang K K, Chin Z Y, Wang C, et al. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci, 2012, 6: 39.
32. Luo T J, Zhou C L, Chao F. Exploring spatial-frequency-sequential relationships for motor imagery classification with recurrent neural network. BMC Bioinformatics, 2018, 19(1): 344-362.
33. Liu C, Jing J, Daly I, et al. SincNet-based hybrid neural network for motor imagery EEG decoding. IEEE Trans Neural Syst Rehabil Eng, 2020, 30: 540-549.
34. Roy A M. Adaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface. Eng Appl Artif Intel, 2022, 116: 105347.

Next Article
Automatic removal algorithm of electrooculographic artifacts in non-invasive brain-computer interface based on independent component analysis

Journal of Biomedical Engineering

Multi-task motor imagery electroencephalogram classification based on adaptive time-frequency common spatial pattern combined with convolutional neural network

Abstract Full text Figures/Tables Video References Cited by

Next Article

Format

Content