Multimodal high-grade glioma semantic segmentation network with multi-scale and multi-attention fusion mechanism_Journal of Biomedical Engineering

Authors：

WU Yuchao ,  LIN Lan , WU Shuicai

Department of Biomedical Engineering, Faculty of Environment and Life Sciences, Beijing University of Technology, Intelligent Physiological Measurement and Clinical Translation, Beijing International Base for Scientific and Technological Cooperation, Beijing 100124, P. R. China;

Corresponding author：

LIN Lan, Email: lanlin@bjut.edu.cn

Keywords：

Multi-scale feature; Attention mechanism; High-grade glioma segmentation; Convolutional neural network

DOI：

10.7507/1001-5515.202103021

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Glioma is a primary brain tumor with high incidence rate. High-grade gliomas (HGG) are those with the highest degree of malignancy and the lowest degree of survival. Surgical resection and postoperative adjuvant chemoradiotherapy are often used in clinical treatment, so accurate segmentation of tumor-related areas is of great significance for the treatment of patients. In order to improve the segmentation accuracy of HGG, this paper proposes a multi-modal glioma semantic segmentation network with multi-scale feature extraction and multi-attention fusion mechanism. The main contributions are, (1) Multi-scale residual structures were used to extract features from multi-modal gliomas magnetic resonance imaging (MRI); (2) Two types of attention modules were used for features aggregating in channel and spatial; (3) In order to improve the segmentation performance of the whole network, the branch classifier was constructed using ensemble learning strategy to adjust and correct the classification results of the backbone classifier. The experimental results showed that the Dice coefficient values of the proposed segmentation method in this article were 0.909 7, 0.877 3 and 0.839 6 for whole tumor, tumor core and enhanced tumor respectively, and the segmentation results had good boundary continuity in the three-dimensional direction. Therefore, the proposed semantic segmentation network has good segmentation performance for high-grade gliomas lesions.

Citation： WU Yuchao, LIN Lan, WU Shuicai. Multimodal high-grade glioma semantic segmentation network with multi-scale and multi-attention fusion mechanism. Journal of Biomedical Engineering, 2022, 39(3): 433-440. doi: 10.7507/1001-5515.202103021 Copy

1.	刘泽亮, 王效春, 张辉, 等. 磁共振扩散成像在脑胶质瘤预后预测的研究进展. 磁共振成像, 2021, 12(1): 77-80.
2.	汪忠, 李军, 刘崎, 等. 基于改进LeNet-5模型的WHO Ⅱ/Ⅲ级脑胶质瘤影像自动分级的临床研究. 临床神经外科杂志, 2021, 18(1): 21-24, 30.
3.	何远秀, 钟文君, 李悦, 等. 复发脑胶质瘤的诊断及治疗进展. 海南医学, 2021, 32(2): 246-249.
4.	吴玉超, 林岚, 王婧璇, 等. 基于卷积神经网络的语义分割在医学图像中的应用. 生物医学工程学杂志, 2020, 37(3): 533-540.
5.	Tiwari A, Srivastava S, Pant M. Brain tumor segmentation and classification from magnetic resonance images: Review of selected methods from 2014 to 2019. Pattern Recogn Lett, 2020, 131: 244-260.
6.	Feng X, Tustison N J, Patel S H, et al. Brain tumor segmentation using an ensemble of 3D U-Nets and overall survival prediction using radiomic features. Front Comput Neurosci, 2020, 14: 1-12.
7.	Wang G, Li W, Vercauteren T, et al. Automatic brain tumor segmentation based on cascaded convolutional neural networks with uncertainty estimation. Front Comput Neurosc, 2019, 13: 56.
8.	Menze B, Reyes M, Van L K, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE T Med Imaging, 2015, 34(10): 1993-2024.
9.	Shanis Z, Gerber S, Gao M, et al. Intramodality domain adaptation using self ensembling and adversarial training// Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data. Cham: Springer, 2019: 28-36.
10.	Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation// 2015 IEEE International Conference on Computer Vision (ICCV). Santiago: IEEE, 2015: 1520-1528.
11.	Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1-9.
12.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 770-778.
13.	林岚, 吴玉超, 王婧璇, 等. 基于卷积神经网络的语义分割技术及其在脑神经影像应用中的研究进展. 北京工业大学学报, 2021, 47(1): 85-92.
14.	Wang F, Jiang M, Qian C, et al. Residual attention network for image classification// 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Honolulu: IEEE, 2017: 3156-3164.
15.	Hu J, Shen L, Sun G. Squeeze-and-excitation networks// 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City: IEEE, 2018: 7132-7141.
16.	Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module// 2018 the European Conference on Computer Vision (ECCV). Munich: Springer, 2018: 3-19.
17.	Veit A, Wilber M, Belongie S. Residual networks behave like ensembles of relatively shallow networks// NISP’16: Proceedings of the 30th International Conference on Neural Information Processing System. Barcelona: Curran Associates Inc., 2016, 29: 550-558.
18.	Cui S, Mao L, Jiang J, et al. Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network. J Healthc Eng, 2018, 2018: 4940593.
19.	Sun L, Zhang S, Chen H, et al. Brain tumor segmentation and survival prediction using multimodal MRI scans with deep learning. Front Neurosci, 2019, 13: 810.
20.	Myronenko A. 3D MRI brain tumor segmentation using autoencoder regularization// International MICCAI Brainlesion Workshop. Cham: Springer, 2018: 311-320.
21.	Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv, 2015: 1511.07122.
22.	Yosinski J, Clune J, Nguyen A, et al. Understanding neural networks through deep visualization. arXiv, 2015: 1506.06579.
23.	Qi Z, Saeed K, Li F. Embedding deep networks into visual explanations. Artif Intell, 2021, 292: 103435.
24.	Yang C, Rangarajan A, Ranka S. Visual explanations from deep 3D convolutional neural networks for Alzheimer’s disease classification. arXiv, 2018: 1803.02544.
25.	Chattopadhay A, Sarkar A, Howlader P, et al. Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks// 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe: IEEE, 2018: 839-847.

1. 刘泽亮, 王效春, 张辉, 等. 磁共振扩散成像在脑胶质瘤预后预测的研究进展. 磁共振成像, 2021, 12(1): 77-80.
2. 汪忠, 李军, 刘崎, 等. 基于改进LeNet-5模型的WHO Ⅱ/Ⅲ级脑胶质瘤影像自动分级的临床研究. 临床神经外科杂志, 2021, 18(1): 21-24, 30.
3. 何远秀, 钟文君, 李悦, 等. 复发脑胶质瘤的诊断及治疗进展. 海南医学, 2021, 32(2): 246-249.
4. 吴玉超, 林岚, 王婧璇, 等. 基于卷积神经网络的语义分割在医学图像中的应用. 生物医学工程学杂志, 2020, 37(3): 533-540.
5. Tiwari A, Srivastava S, Pant M. Brain tumor segmentation and classification from magnetic resonance images: Review of selected methods from 2014 to 2019. Pattern Recogn Lett, 2020, 131: 244-260.
6. Feng X, Tustison N J, Patel S H, et al. Brain tumor segmentation using an ensemble of 3D U-Nets and overall survival prediction using radiomic features. Front Comput Neurosci, 2020, 14: 1-12.
7. Wang G, Li W, Vercauteren T, et al. Automatic brain tumor segmentation based on cascaded convolutional neural networks with uncertainty estimation. Front Comput Neurosc, 2019, 13: 56.
8. Menze B, Reyes M, Van L K, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE T Med Imaging, 2015, 34(10): 1993-2024.
9. Shanis Z, Gerber S, Gao M, et al. Intramodality domain adaptation using self ensembling and adversarial training// Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data. Cham: Springer, 2019: 28-36.
10. Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation// 2015 IEEE International Conference on Computer Vision (ICCV). Santiago: IEEE, 2015: 1520-1528.
11. Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1-9.
12. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 770-778.
13. 林岚, 吴玉超, 王婧璇, 等. 基于卷积神经网络的语义分割技术及其在脑神经影像应用中的研究进展. 北京工业大学学报, 2021, 47(1): 85-92.
14. Wang F, Jiang M, Qian C, et al. Residual attention network for image classification// 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Honolulu: IEEE, 2017: 3156-3164.
15. Hu J, Shen L, Sun G. Squeeze-and-excitation networks// 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City: IEEE, 2018: 7132-7141.
16. Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module// 2018 the European Conference on Computer Vision (ECCV). Munich: Springer, 2018: 3-19.
17. Veit A, Wilber M, Belongie S. Residual networks behave like ensembles of relatively shallow networks// NISP’16: Proceedings of the 30th International Conference on Neural Information Processing System. Barcelona: Curran Associates Inc., 2016, 29: 550-558.
18. Cui S, Mao L, Jiang J, et al. Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network. J Healthc Eng, 2018, 2018: 4940593.
19. Sun L, Zhang S, Chen H, et al. Brain tumor segmentation and survival prediction using multimodal MRI scans with deep learning. Front Neurosci, 2019, 13: 810.
20. Myronenko A. 3D MRI brain tumor segmentation using autoencoder regularization// International MICCAI Brainlesion Workshop. Cham: Springer, 2018: 311-320.
21. Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv, 2015: 1511.07122.
22. Yosinski J, Clune J, Nguyen A, et al. Understanding neural networks through deep visualization. arXiv, 2015: 1506.06579.
23. Qi Z, Saeed K, Li F. Embedding deep networks into visual explanations. Artif Intell, 2021, 292: 103435.
24. Yang C, Rangarajan A, Ranka S. Visual explanations from deep 3D convolutional neural networks for Alzheimer’s disease classification. arXiv, 2018: 1803.02544.
25. Chattopadhay A, Sarkar A, Howlader P, et al. Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks// 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe: IEEE, 2018: 839-847.

Next Article
Segmentation of ground glass pulmonary nodules using full convolution residual network based on atrous spatial pyramid pooling structure and attention mechanism

Journal of Biomedical Engineering

Multimodal high-grade glioma semantic segmentation network with multi-scale and multi-attention fusion mechanism

Abstract Full text Figures/Tables Video References Cited by

Next Article

Format

Content