Key technology of brain-computer interaction based on speech imagery_Journal of Biomedical Engineering

Authors：

LIU Yanpeng ^1,2 , GONG Anmin ³ , DING Peng ^1,2 , ZHAO Lei ⁴ , QIAN Qian ^1,2 , ZHOU Jianhua ^1,2 , SU Lei ^1,2 ,  FU Yunfa ^1,2,5,6

1. School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China;
2. Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China;
3. College of Information Engineering, Engineering University of PAP, Xi’an 710000, P. R. China;
4. Faculty of Science, Kunming University of Science and Technology, Kunming 650500, P. R. China;
5. School of Medicine Brain Science and Visual Cognition Research Center, Kunming University of Science and Technology, Kunming 650500, P. R. China;
6. Yunnan Key Lab of Computer Technology Application, Kunming University of Science and Technology, Kunming 650500, P. R. China;

Corresponding author：

Keywords：

Brain-computer interaction; Speech imagery; Experimental paradigm; Classification; Decoding

DOI：

10.7507/1001-5515.202107018

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Speech expression is an important high-level cognitive behavior of human beings. The realization of this behavior is closely related to human brain activity. Both true speech expression and speech imagination can activate part of the same brain area. Therefore, speech imagery becomes a new paradigm of brain-computer interaction. Brain-computer interface (BCI) based on speech imagery has the advantages of spontaneous generation, no training, and friendliness to subjects, so it has attracted the attention of many scholars. However, this interactive technology is not mature in the design of experimental paradigms and the choice of imagination materials, and there are many issues that need to be discussed urgently. Therefore, in response to these problems, this article first expounds the neural mechanism of speech imagery. Then, by reviewing the previous BCI research of speech imagery, the mainstream methods and core technologies of experimental paradigm, imagination materials, data processing and so on are systematically analyzed. Finally, the key problems and main challenges that restrict the development of this type of BCI are discussed. And the future development and application perspective of the speech imaginary BCI system are prospected.

Citation： LIU Yanpeng, GONG Anmin, DING Peng, ZHAO Lei, QIAN Qian, ZHOU Jianhua, SU Lei, FU Yunfa. Key technology of brain-computer interaction based on speech imagery. Journal of Biomedical Engineering, 2022, 39(3): 596-611. doi: 10.7507/1001-5515.202107018 Copy

1.	Wolpaw J R, Birbaumer N, Mcfarland D J, et al. Brain–computer interfaces for communication and control. Clin Neurophysiol, 2002, 113(6): 767-791.
2.	伏云发, 郭衍龙, 张夏冰, 等. 脑机接口: 变革性的人机交互. 北京: 国防工业出版社, 2020.
3.	李昭阳, 龚安民, 伏云发. 基于EEG脑网络下肢动作视觉想象识别研究. 南京大学学报(自然科学), 2020, 56(4): 570-580.
4.	Yousefi R, Sereshkeh A R, Chau T. Development of a robust asynchronous brain-switch using ErrP-based error correction. J Neural Eng, 2019, 16(6): 066042.
5.	Schafer E W P. Cortical activity preceding speech: Semantic specificity. Nature, 1967, 216(5122): 1338-1339.
6.	Hiraiwa A, Shimohara K, Tokunaga Y. EEG topography recognition by neural networks. IEEE Eng Med Biol, 1990, 9(3): 39-42.
7.	Suppes P, Lu Z L, Han B. Brain wave recognition of words. P Natl Acad Sci USA, 1997, 94(26): 14965-14969.
8.	Brumberg J S, Wright E J, Andreasen D S, et al. Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex. Front Neurosci-Switz, 2011, 5: 00065.
9.	Chaudhary U, Xia B, Silvoni S, et al. Brain–computer interface–based communication in the completely locked-in state. PLoS Biol, 2017, 15(1): e1002593.
10.	陈霏, 潘昌杰. 基于发音想象的脑机接口的研究综述. 信号处理, 2020, 36(6): 816-830.
11.	Schultz T, Wand M, Hueber T, et al. Biosignal-based spoken communication: A survey. IEEE-ACM T Audio Spe, 2017, 25(12): 2257-2271.
12.	Cooney C, Folli R, Coyle D. Neurolinguistics research advancing development of a direct-speech brain-computer interface. iScience, 2018, 8: 103-125.
13.	Martin S, Millan J D R, Knight R T, et al. The use of intracranial recordings to decode human language: Challenges and opportunities. Brain Lang, 2016, 193(2019): 73-83.
14.	Martin S, Iturrate I, Millan J D R, et al. Decoding inner speech using electrocorticography: Progress and challenges toward a speech prosthesis. Front Neurosci-Switz, 2018, 12: 00422.
15.	Panachakel J T, Ramakrishnan A G. Decoding covert speech from EEG-A comprehensive review. Front Neurosci-Switz, 2021, 15: 642251.
16.	Oppenheim G M, Dell G S. Motor movement matters: The flexible abstractness of inner speech. Mem Cognition, 2010, 38(8): 1147-1160.
17.	Palmer E D, Rosen H J, Ojemann J G, et al. An event-related fMRI study of overt and covert word stem completion. Neuroimage, 2001, 14(1): 182-193.
18.	Huang J, Carr T H, Cao Y. Comparing cortical activations for silent and overt speech using event-related fMRI. Hum Brain Mapp, 2002, 15(1): 39-53.
19.	Basho S, Palmer E D, Rubio M A, et al. Effects of generation mode in fMRI adaptations of semantic fluency: Paced production and overt speech. Neuropsychologia, 2007, 45(8): 1697-1706.
20.	Shuster L I, Lemieux S K. An fMRI investigation of covertly and overtly produced mono- and multisyllabic words. Brain Lang, 2005, 93(1): 20-31.
21.	Goto T, Hirata M, Umekawa Y, et al. Frequency-dependent spatiotemporal distribution of cerebral oscillatory changes during silent reading: A magnetoencephalograhic group analysis. Neuroimage, 2011, 54(1): 560-567.
22.	Shergill S, Brammer M, Fukuda R, et al. Modulation of activity in temporal cortex during generation of inner speech. Hum Brain Mapp, 2002, 16(4): 219-227.
23.	Hurlburt R T, Alderson-day B, Kuhn S, et al. Exploring the ecological validity of thinking on demand: Neural correlates of elicited vs. spontaneously occurring inner speech. PloS One, 2016, 11(2): e0147932.
24.	Kellis S, Miller K, Thomson K, et al. Decoding spoken words using local field potentials recorded from the cortical surface. J Neural Eng, 2010, 7(5): 056007.
25.	Leuthardt E C, Gaona C, Sharma M, et al. Using the electrocorticographic speech network to control a brain-computer interface in humans. J Neural Eng, 2011, 8(3): 036004.
26.	Leuthardt E C, Schalk G, Wolpaw J R, et al. A brain-computer interface using electrocorticographic signals in humans. J Neural Eng, 2004, 1(2): 63-71.
27.	Anumanchipalli G K, Chartier J, Chang E F. Speech synthesis from neural decoding of spoken sentences. Nature, 2019, 568(7753): 493-498.
28.	Sun P, Anumanchipalli G K, Chang E F. Brain2Char: a deep architecture for decoding text from brain recordings. J Neural Eng, 2020, 17(6): 066015.
29.	Kaongoen N, Choi J, Jo S. Speech-imagery-based brain-computer interface system using ear-EEG. J Neural Eng, 2021, 18(1): 016023.
30.	Guenther F H, Brumberg J S, Wright E J, et al. A wireless brain-machine interface for real-time speech synthesis. PloS One, 2009, 4(12): e8218.
31.	Pei X, Barbour D, Leuthardt E C, et al. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J Neural Eng, 2011, 8(4): 046028.
32.	Herff C, Heger D, Pesters A D, et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front Neurosci-Switz, 2015, 9: 00217.
33.	Sereshkeh A R, Yousefi R, Wong A T, et al. Online classification of imagined speech using functional near-infrared spectroscopy signals. J Neural Eng, 2018, 16(1): 016005.
34.	Lee S H, Lee M, Jeong J H, et al. Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery// 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). Bari: IEEE, 2019: 4409-4414.
35.	Riaz A, Akhtar S, Iftikhar S, et al. Inter comparison of classification techniques for vowel speech imagery using EEG sensors// The 2014 2nd International Conference on Systems and Informatics (ICSAI 2014). Shanghai: IEEE, 2015: 712-717.
36.	Koizumi, K, Ueda K, Nakao M. Development of a cognitive brain-machine interface based on a visual imagery method// 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu: IEEE, 2018: 1062-1065.
37.	Wang L, Zhang X, Zhong X, et al. Analysis and classification of speech imagery EEG for BCI. Biomed Signal Proces, 2013, 8(6): 901-908.
38.	Jahangiri A, Sepulveda F. The contribution of different frequency bands in class separability of covert speech tasks for BCIs// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju: IEEE, 2017: 2093-2096.
39.	D’Zmura M, Deng S, Lappas T, et al. Toward EEG sensing of imagined speech// Jacko J A. Human-computer interaction. New trends. HCI 2009. Lecture notes in computer science. Berlin, Heidelberg: Springer, 2009: 40-48.
40.	郭苗苗, 齐志光, 王磊, 等. 语言脑机接口康复系统中的参数优化研究. 信号处理, 2018, 34(8): 974-983.
41.	Sereshkeh A R, Trott R, Bricout A, et al. Online EEG classification of covert speech for brain–computer interfacing. Int J Neural Syst, 2017, 27(8): 1750033.
42.	Ikeda S, Shibata T, Nakano N, et al. Neural decoding of single vowels during covert articulation using electrocorticography. Front Hum Neurosci, 2014, 8: 00125.
43.	Crone N E, Hao L, Hart J, et al. Electrocorticographic gamma activity during word production in spoken and sign language. Neurology, 2001, 57(11): 2045-2053.
44.	Lotte F, Brumberg J S, Brunner P, et al. Electrocorticographic representations of segmental features in continuous speech. Front Hum Neurosci, 2015, 9: 00097.
45.	DaSalla C S, Kambara H, Sato M, et al. Single-trial classification of vowel speech imagery using common spatial patterns. Neural Netw, 2009, 22(9): 1334-1339.
46.	杨晓芳, 江铭虎. 基于汉语音位发音想象的脑机接口研究. 中文信息学报, 2014, 28(5): 13-23.
47.	Kim H J, Lee M H, Lee M. A BCI based smart home system combined with event-related potentials and speech imagery task// 2020 8th International Winter Conference on Brain-Computer Interface (BCI). Gangwon: IEEE, 2020: 1-6.
48.	Wang L, Liu X, Liang Z, et al. Analysis and classification of hybrid BCI based on motor imagery and speech imagery. Measurement, 2019, 147: 106842.
49.	Martin S, Brunner P, Iturrate I, et al. Word pair classification during imagined speech using direct brain recordings. Sci Rep-UK, 2016, 6: 25803.
50.	Zhang X, Li H, Chen F. EEG-based classification of imaginary Mandarin tones// 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Montreal: IEEE, 2020: 3889-3892.
51.	Akbari H, Khalighinejad B, Herrero J L, et al. Towards reconstructing intelligible speech from the human auditory cortex. Sci Rep-UK, 2019, 9(1): 874.
52.	Sereshkeh A R, Yousefi R, Wong A T, et al. Development of a ternary hybrid fNIRS-EEG brain–computer interface based on imagined speech. Brain-Computer Interfaces, 2019, 6(2): 1-13.
53.	Chengaiyan S, Retnapandian A S, Anandan K. Identification of vowels in consonant-vowel-consonant words from speech imagery based EEG signals. Cogn Neurodynamics, 2019, 14(1): 1-19.
54.	Qureshi M N I, Min B, Park H J, et al. Multiclass classification of word imagination speech with hybrid connectivity features. IEEE T Bio-Med Eng, 2018, 65(10): 2168-2177.
55.	Mohanchandra K, Saha S. A communication paradigm using subvocalized speech: Translating brain signals into speech. Augment Hum Res, 2016, 1: 3.
56.	Nguyen C H, Karavas G K, Artemiadis P. Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features. J Neural Eng, 2018, 15(1): 016002.
57.	AlSaleh M, Moore R, Christensen H, et al. Discriminating between imagined speech and non-speech tasks using EEG// 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu: IEEE, 2018: 1952-1955.
58.	Coretto G A P, Gareis I E, Rufiner H L. Open access database of EEG signals recorded during imagined speech// 12th International Symposium on Medical Information Processing and Analysis(SIPAIM). Tandil: International Society for Optics and Photonics, 2017: 1016002.
59.	Lee S H, Lee M, Lee S W. EEG representations of spatial and temporal features in imagined speech and overt speech// Palaiahnakote S, Baja G S D, Wang L, et al. Pattern recognition. Aucklannd: Springer, 2020: 387-400.
60.	Song Y, Sepulveda F. A novel onset detection technique for brain–computer interfaces using sound-production related cognitive tasks in simulated-online system. J Neural Eng, 2017, 14(1): 016019.
61.	Song Y, Sepulveda F. Comparison between covert sound-production task (sound-imagery) vs. motor-imagery for onset detection in real-life online self-paced BCIs. J Neuroeng Rehabil, 2020, 17(1): 1-11.
62.	Kumar P, Saini R, Roy P P, et al. Envisioned speech recognition using EEG sensors. Pers Ubiquit Comput, 2018, 22(1): 185-199.
63.	Dash D, Ferrari P, Heitzman D, et al. Decoding speech from single trial MEG signals using convolutional neural networks and transfer learning// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Berlin: IEEE, 2019: 5531-5535.
64.	Dash D, Ferrari P, Wang J. Decoding imagined and spoken phrases from non-invasive neural (MEG) signals. Front Neurosci-Switz, 2020, 14: 00290.
65.	Makin J G, Moses D A, Chang E F. Machine translation of cortical activity to text with an encoder-decoder framework. Nat Neurosci, 2020, 23(4): 575-582.
66.	Zhao S, Rudzicz F. Classifying phonological categories in imagined and articulated speech// 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). South Brisbane: IEEE, 2015: 992-996.
67.	Min B, Kim J, Park H J, et al. Vowel imagery decoding toward silent speech BCI using extreme learning machine with electroencephalogram. Biomed Res Int, 2016, 2016: 2618265.
68.	Hwang H J, Choi H, Kim J Y, et al. Toward more intuitive brain–computer interfacing: classification of binary covert intentions using functional near-infrared spectroscopy. J Biomed Opt, 2016, 21(9): 091303.
69.	Iqbal S, Shanir P P M, Khan Y U, et al. Time domain analysis of EEG to classify imagined speech// Satapathy S, Raju K, Mandal J, et al. Proceedings of the Second International Conference on Computer and Communication Technologies. Advances in intelligent systems and computing. New Delhi: Springer, 2016: 793-800.
70.	Tottrup L, Leerskov K, Hadsund J T, et al. Decoding covert speech for intuitive control of brain-computer interfaces based on single-trial EEG: a feasibility study// 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR). Toronto: IEEE, 2019: 689-693.
71.	Sereshkeh A R, Trott R, Bricout A, et al. EEG classification of covert speech using regularized neural networks. IEEE-ACM T Audio Spe, 2017, 25(12): 2292-2300.
72.	Hashim N, Ali A, Mohd-Isa W N. Word-based classification of imagined speech using EEG// Alfred R, Iida H, Ag I A, et al. Computational science and technology. Singapore: Springer, 2018: 195-204.
73.	Muda L, Begam M, Elamvazuthi I. Voice recognition algorithms using Mel Frequency Cepstral Coefficient (MFCC) and Dynamic Time Warping (DTW) techniques. J Comput, 2010, 2(3): 138-143.
74.	Cooney C, Folli R, Coyle D. Mel Frequency Cepstral Coefficients enhance imagined speech decoding from EEG// 2018 29th Irish Signals and Systems Conference (ISSC). Belfast: IEEE, 2018: 1-7.
75.	Lee S H, Lee M, Lee S W. Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication. IEEE T Neur Sys Reh, 2020, 28(12): 2647-2659.
76.	Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Proc Mag, 2008, 25(1): 41-56.
77.	Garcia-Salinas J S, Villasenor-Pineda L, Reyes-Garcia C A, et al. Tensor decomposition for imagined speech discrimination in EEG// Batyrshin I, Martinez-Villasenor M, Ponce E H. Advances in computational intelligence. Guadalajara: Springer, 2018: 239-249.
78.	Bakhshali M A, Khademi M, Ebrahimi-Moghadam A, et al. EEG signal classification of imagined speech based on Riemannian distance of correntropy spectral density. Biomed Signal Proces and Control, 2020, 59: 101899.
79.	Yoshimura N, Nishimoto A, Belkacem A N, et al. Decoding of covert vowel articulation using electroencephalography cortical currents. Front Neurosci-Switz, 2016, 10: 00175.
80.	Jahangiri A, Achanccaray D, Sepulveda F. A novel EEG-based four-class linguistic BCI// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Berlin: IEEE, 2019: 3050-3053.
81.	Deng S, Srinivasan R, Lappas T, et al. EEG classification of imagined syllable rhythm using Hilbert spectrum methods. J Neural Eng, 2010, 7(4): 046006.
82.	Pawar D, Dhage S. Multiclass covert speech classification using extreme learning machine. Biomed Eng Lett, 2020, 10(2): 217-226.
83.	Sarmiento L C, Cortes C J, Bacca J A, et al. Brain computer interface (BCI) with EEG signals for automatic vowel recognition based on articulation mode// 5th ISSNIP-IEEE Biosignals and Biorobotics Conference (2014): Biosignals and Robotics for Better and Safer Living (BRC). Salvador: IEEE, 2014: 1-4.
84.	Torres-Garcia A A, Reyes-Garcia C A, Villasenor-Pineda L. Toward a silent speech interface based on unspoken speech//Proceedings of Biosignals 2012 (BIOSTEC). Algarve: SciTePress, 2012: 370-373.
85.	Torres-Garcia A A, Reyes-Garcia C A, Villasenor-Pineda L, et al. Implementing a fuzzy inference system in a multi-objective EEG channel selection model for imagined speech classification. Expert Syst Appl, 2016, 59: 1-12.
86.	Matsumoto M, Hori J. Classification of silent speech using support vector machine and relevance vector machine. Appl Soft Comput, 2014, 20: 95-102.
87.	Cooney C, Folli R, Coyle D. Optimizing input layers improves CNN generalization and transfer learning for imagined speech decoding from EEG// 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). Bari: IEEE, 2019: 1311-1316.
88.	Jimenez-Guarneros M, Gomez-Gil P. Standardization-refinement domain adaptation method for cross-subject EEG-based classification in imagined speech recognition. Pattern Recogn Lett, 2021, 141: 54-60.
89.	Saha P, Fels S, Abdul-Mageed M. Deep learning the EEG manifold for phonological categorization from active thoughts// 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton: IEEE, 2019: 2762-2766.
90.	Cooney C, Korik A, Folli R, et al. Classification of imagined spoken word-pairs using convolutional neural networks// Gernot R M P, Jonas C D, Selina C W. Proceedings of the 8th Graz Brain Computer Interface Conference 2019. Graz: Verlag der Technischen Universitat Graz, 2019: 338-343.
91.	Parhi M, Tewfik A H. Classifying imaginary vowels from frontal lobe EEG via deep learning// 2020 28th European Signal Processing Conference (EUSIPCO). Amsterdam: IEEE, 2021: 1195-1199.
92.	Saha P, Fels S. Hierarchical deep feature learning for decoding imagined speech from EEG// Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu: AAAI, 2019: 10019-10020.
93.	Panachakel J T, Ramakrishnan A G, Ananthapadmanabha T V. A novel deep learning architecture for decoding imagined speech from EEG. arXiv preprint arXiv, 2020: 2003.09374.
94.	Torres J M M, Stepanov E A, Riccardi G. EEG semantic decoding using deep neural networks// Concepts, Actions and Objects Workshop CAOs 2016. Rovereto: Personal Healthcare Agents, 2016: 1-2.
95.	Okhovvat M, Sharifi M, Bidgoli B M. An accurate Persian part-of-speech tagger. Comput Syst Sci Eng, 2020, 35(6): 423-430.
96.	Sun P, Qin J. Neural networks based EEG-speech models. arXiv preprint arXiv, 2016: 1612.05369.
97.	Panachakel J T, Ramakrishnan A G, Ananthapadmanabha T V. Decoding imagined speech using wavelet features and deep neural networks// 2019 IEEE 16th India Council International Conference (INDICON). Rajkot: IEEE, 2019: 1-4.
98.	Cooney C, Korik A, Folli R, et al. Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech EEG. Sensors, 2020, 20(16): 4629.
99.	Matsumoto M. Silent speech decoder using adaptive collection// Kuflik T, Stock O. Proceedings of the Companion Publication of the 19th International Conference on Intelligent User Interfaces. New York: Association for Computing Machinery, 2014: 73-76.
100.	Iqbal S, Khan Y U, Farooq O. EEG based classification of imagined vowel sounds// 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom). New Delhi: IEEE, 2015: 1591-1594.
101.	吕晓彤, 丁鹏, 李思语, 等. 脑机接口人因工程及应用:以人为中心的脑机接口设计和评价方法. 生物医学工程学杂志, 2021, 38(2): 210-223.
102.	Wang L, Huang W, Yang Z, et al. A method from offline analysis to online training for the brain-computer interface based on motor imagery and speech imagery. Biomed Signal Proces, 2020, 62: 102100.
103.	熊馨, 杨秋红, 周建华, 等. 脑机融合控制中脑电伪迹处理方法. 昆明理工大学学报(自然科学版), 2021, 46(3): 56-70.
104.	Wan Z, Yang R, Huang M, et al. A review on transfer learning in EEG signal analysis. Neurocomputing, 2021, 421: 1-14.
105.	Herff C, Heger D, Putze F, et al. Cross-subject classification of speaking modes using fnirs// International Conference on Neural Information Processing. Berlin, Heidelberg: Springer, 2012: 417-424.
106.	Wang L, Zhang X, Zhong X, et al. Improvement of mental tasks with relevant speech imagery for brain-computer interfaces. Measurement, 2016, 91: 201-209.
107.	Denby B, Schultz T, Honda K, et al. Silent speech interfaces. Speech Commun, 2010, 52(4): 270-287.
108.	伏云发, 龚安民, 南文雅. 神经反馈原理与实践. 北京: 电子工业出版社, 2021.
109.	汪露雲, 孔万增, 张昕昱, 等. 脑纹识别研究综述. 中国生物医学工程学报, 2017, 36(5): 602-607.
110.	Brigham K, Kumar B V K V. Subject identification from electroencephalogram (EEG) signals during imagined speech// 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS). Washington: IEEE, 2010: 1-8.
111.	伏云发, 龚安民, 陈超, 等. 面向实用的脑-机接口: 缩小研究与实际应用之间的差距. 北京: 科学出版社, 2022.
112.	伏云发, 杨秋红, 徐宝磊, 等. 脑机接口原理与实践. 北京: 国防工业出版社, 2017.
113.	罗建功, 丁鹏, 龚安民, 等. 脑机接口技术的应用、产业转化和商业价值. 生物医学工程学杂志, 2022, 39(2): 405-415.

1. Wolpaw J R, Birbaumer N, Mcfarland D J, et al. Brain–computer interfaces for communication and control. Clin Neurophysiol, 2002, 113(6): 767-791.
2. 伏云发, 郭衍龙, 张夏冰, 等. 脑机接口: 变革性的人机交互. 北京: 国防工业出版社, 2020.
3. 李昭阳, 龚安民, 伏云发. 基于EEG脑网络下肢动作视觉想象识别研究. 南京大学学报(自然科学), 2020, 56(4): 570-580.
4. Yousefi R, Sereshkeh A R, Chau T. Development of a robust asynchronous brain-switch using ErrP-based error correction. J Neural Eng, 2019, 16(6): 066042.
5. Schafer E W P. Cortical activity preceding speech: Semantic specificity. Nature, 1967, 216(5122): 1338-1339.
6. Hiraiwa A, Shimohara K, Tokunaga Y. EEG topography recognition by neural networks. IEEE Eng Med Biol, 1990, 9(3): 39-42.
7. Suppes P, Lu Z L, Han B. Brain wave recognition of words. P Natl Acad Sci USA, 1997, 94(26): 14965-14969.
8. Brumberg J S, Wright E J, Andreasen D S, et al. Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex. Front Neurosci-Switz, 2011, 5: 00065.
9. Chaudhary U, Xia B, Silvoni S, et al. Brain–computer interface–based communication in the completely locked-in state. PLoS Biol, 2017, 15(1): e1002593.
10. 陈霏, 潘昌杰. 基于发音想象的脑机接口的研究综述. 信号处理, 2020, 36(6): 816-830.
11. Schultz T, Wand M, Hueber T, et al. Biosignal-based spoken communication: A survey. IEEE-ACM T Audio Spe, 2017, 25(12): 2257-2271.
12. Cooney C, Folli R, Coyle D. Neurolinguistics research advancing development of a direct-speech brain-computer interface. iScience, 2018, 8: 103-125.
13. Martin S, Millan J D R, Knight R T, et al. The use of intracranial recordings to decode human language: Challenges and opportunities. Brain Lang, 2016, 193(2019): 73-83.
14. Martin S, Iturrate I, Millan J D R, et al. Decoding inner speech using electrocorticography: Progress and challenges toward a speech prosthesis. Front Neurosci-Switz, 2018, 12: 00422.
15. Panachakel J T, Ramakrishnan A G. Decoding covert speech from EEG-A comprehensive review. Front Neurosci-Switz, 2021, 15: 642251.
16. Oppenheim G M, Dell G S. Motor movement matters: The flexible abstractness of inner speech. Mem Cognition, 2010, 38(8): 1147-1160.
17. Palmer E D, Rosen H J, Ojemann J G, et al. An event-related fMRI study of overt and covert word stem completion. Neuroimage, 2001, 14(1): 182-193.
18. Huang J, Carr T H, Cao Y. Comparing cortical activations for silent and overt speech using event-related fMRI. Hum Brain Mapp, 2002, 15(1): 39-53.
19. Basho S, Palmer E D, Rubio M A, et al. Effects of generation mode in fMRI adaptations of semantic fluency: Paced production and overt speech. Neuropsychologia, 2007, 45(8): 1697-1706.
20. Shuster L I, Lemieux S K. An fMRI investigation of covertly and overtly produced mono- and multisyllabic words. Brain Lang, 2005, 93(1): 20-31.
21. Goto T, Hirata M, Umekawa Y, et al. Frequency-dependent spatiotemporal distribution of cerebral oscillatory changes during silent reading: A magnetoencephalograhic group analysis. Neuroimage, 2011, 54(1): 560-567.
22. Shergill S, Brammer M, Fukuda R, et al. Modulation of activity in temporal cortex during generation of inner speech. Hum Brain Mapp, 2002, 16(4): 219-227.
23. Hurlburt R T, Alderson-day B, Kuhn S, et al. Exploring the ecological validity of thinking on demand: Neural correlates of elicited vs. spontaneously occurring inner speech. PloS One, 2016, 11(2): e0147932.
24. Kellis S, Miller K, Thomson K, et al. Decoding spoken words using local field potentials recorded from the cortical surface. J Neural Eng, 2010, 7(5): 056007.
25. Leuthardt E C, Gaona C, Sharma M, et al. Using the electrocorticographic speech network to control a brain-computer interface in humans. J Neural Eng, 2011, 8(3): 036004.
26. Leuthardt E C, Schalk G, Wolpaw J R, et al. A brain-computer interface using electrocorticographic signals in humans. J Neural Eng, 2004, 1(2): 63-71.
27. Anumanchipalli G K, Chartier J, Chang E F. Speech synthesis from neural decoding of spoken sentences. Nature, 2019, 568(7753): 493-498.
28. Sun P, Anumanchipalli G K, Chang E F. Brain2Char: a deep architecture for decoding text from brain recordings. J Neural Eng, 2020, 17(6): 066015.
29. Kaongoen N, Choi J, Jo S. Speech-imagery-based brain-computer interface system using ear-EEG. J Neural Eng, 2021, 18(1): 016023.
30. Guenther F H, Brumberg J S, Wright E J, et al. A wireless brain-machine interface for real-time speech synthesis. PloS One, 2009, 4(12): e8218.
31. Pei X, Barbour D, Leuthardt E C, et al. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J Neural Eng, 2011, 8(4): 046028.
32. Herff C, Heger D, Pesters A D, et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front Neurosci-Switz, 2015, 9: 00217.
33. Sereshkeh A R, Yousefi R, Wong A T, et al. Online classification of imagined speech using functional near-infrared spectroscopy signals. J Neural Eng, 2018, 16(1): 016005.
34. Lee S H, Lee M, Jeong J H, et al. Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery// 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). Bari: IEEE, 2019: 4409-4414.
35. Riaz A, Akhtar S, Iftikhar S, et al. Inter comparison of classification techniques for vowel speech imagery using EEG sensors// The 2014 2nd International Conference on Systems and Informatics (ICSAI 2014). Shanghai: IEEE, 2015: 712-717.
36. Koizumi, K, Ueda K, Nakao M. Development of a cognitive brain-machine interface based on a visual imagery method// 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu: IEEE, 2018: 1062-1065.
37. Wang L, Zhang X, Zhong X, et al. Analysis and classification of speech imagery EEG for BCI. Biomed Signal Proces, 2013, 8(6): 901-908.
38. Jahangiri A, Sepulveda F. The contribution of different frequency bands in class separability of covert speech tasks for BCIs// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju: IEEE, 2017: 2093-2096.
39. D’Zmura M, Deng S, Lappas T, et al. Toward EEG sensing of imagined speech// Jacko J A. Human-computer interaction. New trends. HCI 2009. Lecture notes in computer science. Berlin, Heidelberg: Springer, 2009: 40-48.
40. 郭苗苗, 齐志光, 王磊, 等. 语言脑机接口康复系统中的参数优化研究. 信号处理, 2018, 34(8): 974-983.
41. Sereshkeh A R, Trott R, Bricout A, et al. Online EEG classification of covert speech for brain–computer interfacing. Int J Neural Syst, 2017, 27(8): 1750033.
42. Ikeda S, Shibata T, Nakano N, et al. Neural decoding of single vowels during covert articulation using electrocorticography. Front Hum Neurosci, 2014, 8: 00125.
43. Crone N E, Hao L, Hart J, et al. Electrocorticographic gamma activity during word production in spoken and sign language. Neurology, 2001, 57(11): 2045-2053.
44. Lotte F, Brumberg J S, Brunner P, et al. Electrocorticographic representations of segmental features in continuous speech. Front Hum Neurosci, 2015, 9: 00097.
45. DaSalla C S, Kambara H, Sato M, et al. Single-trial classification of vowel speech imagery using common spatial patterns. Neural Netw, 2009, 22(9): 1334-1339.
46. 杨晓芳, 江铭虎. 基于汉语音位发音想象的脑机接口研究. 中文信息学报, 2014, 28(5): 13-23.
47. Kim H J, Lee M H, Lee M. A BCI based smart home system combined with event-related potentials and speech imagery task// 2020 8th International Winter Conference on Brain-Computer Interface (BCI). Gangwon: IEEE, 2020: 1-6.
48. Wang L, Liu X, Liang Z, et al. Analysis and classification of hybrid BCI based on motor imagery and speech imagery. Measurement, 2019, 147: 106842.
49. Martin S, Brunner P, Iturrate I, et al. Word pair classification during imagined speech using direct brain recordings. Sci Rep-UK, 2016, 6: 25803.
50. Zhang X, Li H, Chen F. EEG-based classification of imaginary Mandarin tones// 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Montreal: IEEE, 2020: 3889-3892.
51. Akbari H, Khalighinejad B, Herrero J L, et al. Towards reconstructing intelligible speech from the human auditory cortex. Sci Rep-UK, 2019, 9(1): 874.
52. Sereshkeh A R, Yousefi R, Wong A T, et al. Development of a ternary hybrid fNIRS-EEG brain–computer interface based on imagined speech. Brain-Computer Interfaces, 2019, 6(2): 1-13.
53. Chengaiyan S, Retnapandian A S, Anandan K. Identification of vowels in consonant-vowel-consonant words from speech imagery based EEG signals. Cogn Neurodynamics, 2019, 14(1): 1-19.
54. Qureshi M N I, Min B, Park H J, et al. Multiclass classification of word imagination speech with hybrid connectivity features. IEEE T Bio-Med Eng, 2018, 65(10): 2168-2177.
55. Mohanchandra K, Saha S. A communication paradigm using subvocalized speech: Translating brain signals into speech. Augment Hum Res, 2016, 1: 3.
56. Nguyen C H, Karavas G K, Artemiadis P. Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features. J Neural Eng, 2018, 15(1): 016002.
57. AlSaleh M, Moore R, Christensen H, et al. Discriminating between imagined speech and non-speech tasks using EEG// 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu: IEEE, 2018: 1952-1955.
58. Coretto G A P, Gareis I E, Rufiner H L. Open access database of EEG signals recorded during imagined speech// 12th International Symposium on Medical Information Processing and Analysis(SIPAIM). Tandil: International Society for Optics and Photonics, 2017: 1016002.
59. Lee S H, Lee M, Lee S W. EEG representations of spatial and temporal features in imagined speech and overt speech// Palaiahnakote S, Baja G S D, Wang L, et al. Pattern recognition. Aucklannd: Springer, 2020: 387-400.
60. Song Y, Sepulveda F. A novel onset detection technique for brain–computer interfaces using sound-production related cognitive tasks in simulated-online system. J Neural Eng, 2017, 14(1): 016019.
61. Song Y, Sepulveda F. Comparison between covert sound-production task (sound-imagery) vs. motor-imagery for onset detection in real-life online self-paced BCIs. J Neuroeng Rehabil, 2020, 17(1): 1-11.
62. Kumar P, Saini R, Roy P P, et al. Envisioned speech recognition using EEG sensors. Pers Ubiquit Comput, 2018, 22(1): 185-199.
63. Dash D, Ferrari P, Heitzman D, et al. Decoding speech from single trial MEG signals using convolutional neural networks and transfer learning// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Berlin: IEEE, 2019: 5531-5535.
64. Dash D, Ferrari P, Wang J. Decoding imagined and spoken phrases from non-invasive neural (MEG) signals. Front Neurosci-Switz, 2020, 14: 00290.
65. Makin J G, Moses D A, Chang E F. Machine translation of cortical activity to text with an encoder-decoder framework. Nat Neurosci, 2020, 23(4): 575-582.
66. Zhao S, Rudzicz F. Classifying phonological categories in imagined and articulated speech// 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). South Brisbane: IEEE, 2015: 992-996.
67. Min B, Kim J, Park H J, et al. Vowel imagery decoding toward silent speech BCI using extreme learning machine with electroencephalogram. Biomed Res Int, 2016, 2016: 2618265.
68. Hwang H J, Choi H, Kim J Y, et al. Toward more intuitive brain–computer interfacing: classification of binary covert intentions using functional near-infrared spectroscopy. J Biomed Opt, 2016, 21(9): 091303.
69. Iqbal S, Shanir P P M, Khan Y U, et al. Time domain analysis of EEG to classify imagined speech// Satapathy S, Raju K, Mandal J, et al. Proceedings of the Second International Conference on Computer and Communication Technologies. Advances in intelligent systems and computing. New Delhi: Springer, 2016: 793-800.
70. Tottrup L, Leerskov K, Hadsund J T, et al. Decoding covert speech for intuitive control of brain-computer interfaces based on single-trial EEG: a feasibility study// 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR). Toronto: IEEE, 2019: 689-693.
71. Sereshkeh A R, Trott R, Bricout A, et al. EEG classification of covert speech using regularized neural networks. IEEE-ACM T Audio Spe, 2017, 25(12): 2292-2300.
72. Hashim N, Ali A, Mohd-Isa W N. Word-based classification of imagined speech using EEG// Alfred R, Iida H, Ag I A, et al. Computational science and technology. Singapore: Springer, 2018: 195-204.
73. Muda L, Begam M, Elamvazuthi I. Voice recognition algorithms using Mel Frequency Cepstral Coefficient (MFCC) and Dynamic Time Warping (DTW) techniques. J Comput, 2010, 2(3): 138-143.
74. Cooney C, Folli R, Coyle D. Mel Frequency Cepstral Coefficients enhance imagined speech decoding from EEG// 2018 29th Irish Signals and Systems Conference (ISSC). Belfast: IEEE, 2018: 1-7.
75. Lee S H, Lee M, Lee S W. Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication. IEEE T Neur Sys Reh, 2020, 28(12): 2647-2659.
76. Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Proc Mag, 2008, 25(1): 41-56.
77. Garcia-Salinas J S, Villasenor-Pineda L, Reyes-Garcia C A, et al. Tensor decomposition for imagined speech discrimination in EEG// Batyrshin I, Martinez-Villasenor M, Ponce E H. Advances in computational intelligence. Guadalajara: Springer, 2018: 239-249.
78. Bakhshali M A, Khademi M, Ebrahimi-Moghadam A, et al. EEG signal classification of imagined speech based on Riemannian distance of correntropy spectral density. Biomed Signal Proces and Control, 2020, 59: 101899.
79. Yoshimura N, Nishimoto A, Belkacem A N, et al. Decoding of covert vowel articulation using electroencephalography cortical currents. Front Neurosci-Switz, 2016, 10: 00175.
80. Jahangiri A, Achanccaray D, Sepulveda F. A novel EEG-based four-class linguistic BCI// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Berlin: IEEE, 2019: 3050-3053.
81. Deng S, Srinivasan R, Lappas T, et al. EEG classification of imagined syllable rhythm using Hilbert spectrum methods. J Neural Eng, 2010, 7(4): 046006.
82. Pawar D, Dhage S. Multiclass covert speech classification using extreme learning machine. Biomed Eng Lett, 2020, 10(2): 217-226.
83. Sarmiento L C, Cortes C J, Bacca J A, et al. Brain computer interface (BCI) with EEG signals for automatic vowel recognition based on articulation mode// 5th ISSNIP-IEEE Biosignals and Biorobotics Conference (2014): Biosignals and Robotics for Better and Safer Living (BRC). Salvador: IEEE, 2014: 1-4.
84. Torres-Garcia A A, Reyes-Garcia C A, Villasenor-Pineda L. Toward a silent speech interface based on unspoken speech//Proceedings of Biosignals 2012 (BIOSTEC). Algarve: SciTePress, 2012: 370-373.
85. Torres-Garcia A A, Reyes-Garcia C A, Villasenor-Pineda L, et al. Implementing a fuzzy inference system in a multi-objective EEG channel selection model for imagined speech classification. Expert Syst Appl, 2016, 59: 1-12.
86. Matsumoto M, Hori J. Classification of silent speech using support vector machine and relevance vector machine. Appl Soft Comput, 2014, 20: 95-102.
87. Cooney C, Folli R, Coyle D. Optimizing input layers improves CNN generalization and transfer learning for imagined speech decoding from EEG// 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). Bari: IEEE, 2019: 1311-1316.
88. Jimenez-Guarneros M, Gomez-Gil P. Standardization-refinement domain adaptation method for cross-subject EEG-based classification in imagined speech recognition. Pattern Recogn Lett, 2021, 141: 54-60.
89. Saha P, Fels S, Abdul-Mageed M. Deep learning the EEG manifold for phonological categorization from active thoughts// 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton: IEEE, 2019: 2762-2766.
90. Cooney C, Korik A, Folli R, et al. Classification of imagined spoken word-pairs using convolutional neural networks// Gernot R M P, Jonas C D, Selina C W. Proceedings of the 8th Graz Brain Computer Interface Conference 2019. Graz: Verlag der Technischen Universitat Graz, 2019: 338-343.
91. Parhi M, Tewfik A H. Classifying imaginary vowels from frontal lobe EEG via deep learning// 2020 28th European Signal Processing Conference (EUSIPCO). Amsterdam: IEEE, 2021: 1195-1199.
92. Saha P, Fels S. Hierarchical deep feature learning for decoding imagined speech from EEG// Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu: AAAI, 2019: 10019-10020.
93. Panachakel J T, Ramakrishnan A G, Ananthapadmanabha T V. A novel deep learning architecture for decoding imagined speech from EEG. arXiv preprint arXiv, 2020: 2003.09374.
94. Torres J M M, Stepanov E A, Riccardi G. EEG semantic decoding using deep neural networks// Concepts, Actions and Objects Workshop CAOs 2016. Rovereto: Personal Healthcare Agents, 2016: 1-2.
95. Okhovvat M, Sharifi M, Bidgoli B M. An accurate Persian part-of-speech tagger. Comput Syst Sci Eng, 2020, 35(6): 423-430.
96. Sun P, Qin J. Neural networks based EEG-speech models. arXiv preprint arXiv, 2016: 1612.05369.
97. Panachakel J T, Ramakrishnan A G, Ananthapadmanabha T V. Decoding imagined speech using wavelet features and deep neural networks// 2019 IEEE 16th India Council International Conference (INDICON). Rajkot: IEEE, 2019: 1-4.
98. Cooney C, Korik A, Folli R, et al. Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech EEG. Sensors, 2020, 20(16): 4629.
99. Matsumoto M. Silent speech decoder using adaptive collection// Kuflik T, Stock O. Proceedings of the Companion Publication of the 19th International Conference on Intelligent User Interfaces. New York: Association for Computing Machinery, 2014: 73-76.
100. Iqbal S, Khan Y U, Farooq O. EEG based classification of imagined vowel sounds// 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom). New Delhi: IEEE, 2015: 1591-1594.
101. 吕晓彤, 丁鹏, 李思语, 等. 脑机接口人因工程及应用:以人为中心的脑机接口设计和评价方法. 生物医学工程学杂志, 2021, 38(2): 210-223.
102. Wang L, Huang W, Yang Z, et al. A method from offline analysis to online training for the brain-computer interface based on motor imagery and speech imagery. Biomed Signal Proces, 2020, 62: 102100.
103. 熊馨, 杨秋红, 周建华, 等. 脑机融合控制中脑电伪迹处理方法. 昆明理工大学学报(自然科学版), 2021, 46(3): 56-70.
104. Wan Z, Yang R, Huang M, et al. A review on transfer learning in EEG signal analysis. Neurocomputing, 2021, 421: 1-14.
105. Herff C, Heger D, Putze F, et al. Cross-subject classification of speaking modes using fnirs// International Conference on Neural Information Processing. Berlin, Heidelberg: Springer, 2012: 417-424.
106. Wang L, Zhang X, Zhong X, et al. Improvement of mental tasks with relevant speech imagery for brain-computer interfaces. Measurement, 2016, 91: 201-209.
107. Denby B, Schultz T, Honda K, et al. Silent speech interfaces. Speech Commun, 2010, 52(4): 270-287.
108. 伏云发, 龚安民, 南文雅. 神经反馈原理与实践. 北京: 电子工业出版社, 2021.
109. 汪露雲, 孔万增, 张昕昱, 等. 脑纹识别研究综述. 中国生物医学工程学报, 2017, 36(5): 602-607.
110. Brigham K, Kumar B V K V. Subject identification from electroencephalogram (EEG) signals during imagined speech// 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS). Washington: IEEE, 2010: 1-8.
111. 伏云发, 龚安民, 陈超, 等. 面向实用的脑-机接口: 缩小研究与实际应用之间的差距. 北京: 科学出版社, 2022.
112. 伏云发, 杨秋红, 徐宝磊, 等. 脑机接口原理与实践. 北京: 国防工业出版社, 2017.
113. 罗建功, 丁鹏, 龚安民, 等. 脑机接口技术的应用、产业转化和商业价值. 生物医学工程学杂志, 2022, 39(2): 405-415.

Previous Article
Intelligent fault diagnosis expert system for multi-parameter monitor based on fault tree
Next Article
Research progress and application of transfer entropy algorithm

Journal of Biomedical Engineering

Key technology of brain-computer interaction based on speech imagery

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content