Deep learning method for magnetic resonance imaging fluid-attenuated inversion recovery image synthesis_Journal of Biomedical Engineering

Authors：

ZHOU Jianing ¹ ,  GUO Hongyu ^1,2 , CHEN Hong ¹

1. School of Electrical Engineering, Shenyang University of Technology, Shenyang 110870, P. R. China;
2. Neusoft Medical System Co. Ltd, Shenyang 110167, P. R. China;

Corresponding author：

GUO Hongyu, Email: guohongyu@sut.edu.cn

Keywords：

Magnetic resonance imaging; Deep learning; Multi-modal; Feature fusion

DOI：

10.7507/1001-5515.202302012

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Magnetic resonance imaging(MRI) can obtain multi-modal images with different contrast, which provides rich information for clinical diagnosis. However, some contrast images are not scanned or the quality of the acquired images cannot meet the diagnostic requirements due to the difficulty of patient's cooperation or the limitation of scanning conditions. Image synthesis techniques have become a method to compensate for such image deficiencies. In recent years, deep learning has been widely used in the field of MRI synthesis. In this paper, a synthesis network based on multi-modal fusion is proposed, which firstly uses a feature encoder to encode the features of multiple unimodal images separately, and then fuses the features of different modal images through a feature fusion module, and finally generates the target modal image. The similarity measure between the target image and the predicted image in the network is improved by introducing a dynamic weighted combined loss function based on the spatial domain and K-space domain. After experimental validation and quantitative comparison, the multi-modal fusion deep learning network proposed in this paper can effectively synthesize high-quality MRI fluid-attenuated inversion recovery (FLAIR) images. In summary, the method proposed in this paper can reduce MRI scanning time of the patient, as well as solve the clinical problem of missing FLAIR images or image quality that is difficult to meet diagnostic requirements.

Citation： ZHOU Jianing, GUO Hongyu, CHEN Hong. Deep learning method for magnetic resonance imaging fluid-attenuated inversion recovery image synthesis. Journal of Biomedical Engineering, 2023, 40(5): 903-911. doi: 10.7507/1001-5515.202302012 Copy

1.	Iglesias J E, Konukoglu E, Zikic D, et al. Is synthesizing MRI contrast useful for inter-modality analysis?. Med Image Comput Comput Assist Interv, 2013, 16(Pt 1): 631-638.
2.	Lustig M, Donoho D, Pauly J M. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 2007, 58(6): 1182-1195.
3.	Zhou T, Fu H, Chen G, et al. Hi-Net: hybrid-fusion network for multi-modal MR image synthesis. IEEE Transactions on Medical Imaging, 2022, 39(9): 2772-2781.
4.	Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks//2017 IEEE International Conference on Computer Vision(ICCV). Venice: University of Venice, 2017: 2242-2251.
5.	Galbusera F, Bassani T, Casaroli G, et al. Generative models: an upcoming innovation in musculoskeletal radiology? A preliminary test in spine imaging. European Radiology Experimental, 2018, 2(1): 29-33.
6.	楼鑫杰. 基于深度学习与多模态融合的快速MRI重建研究. 浙江: 浙江工业大学, 2021.
7.	Yang X, Lin Y, Wang Z, et al. Bi-modality medical image synthesis using semi-supervised sequential generative adversarial networks. IEEE Journal of Biomedical and Health Informatics, 2020, 24(3): 855-865.
8.	Kim S, Jang H, Jang J, et al. Deep-learned short tau inversion recovery imaging using multi-contrast MR images. Magnetic Resonance in Medicine, 2020, 84(6): 2994-3008.
9.	Torrado-Carvajal A, Herraiz J L, Alcain E, et al. Fast patch-based pseudo-CT synthesis from T1-weighted MR images for PET/MR attenuation correction in brain studies. The Journal of Nuclear Medicine, 2016, 57(1): 136-143.
10.	Havaei M, Guizard N, Chapados N, et al. HeMIS: hetero-modal image segmentation //Medical Image Computing and Computer-Assisted Intervention (MICCAI). Athens: Imagia Inc, 2016: 469-477.
11.	Fei Y, Zhan B, Hong M, et al. Deep learning-based multi-modal computing with feature disentanglement for MRI image synthesis. Medical Physics, 2021, 48(7): 3378-3789.
12.	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets//The International Conference on Neural Information Processing Systems. Montreal: NIPS, 2014: 2672-2680.
13.	Baid U, Ghodasara S, Mohan S, et al. The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint, 2021, arXiv: 2107.02314.
14.	Menze B H, Jakab A, Bauer S, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 2015, 34(10): 1993-2024.
15.	Bakas S, Reyes M, Jakab A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint, 2021, arXiv: 1811.02629.
16.	Feng C M, Yan Y, Liu C, et al. Exploring separable attention for multi-contrast MR image super-resolution. arXiv preprint, 2021, arXiv: 2109.01664.
17.	李英. 基于结构相似性和低秩稀疏的磁共振图像去噪算法研究. 计算机应用, 2017, 37(5): 1375-1379.
18.	Dvorák P, Menze B. Structured prediction with convolutional neural networks for multimodal brain tumor segmentation// MICCAI Multimodal Brain Tumor Segmentation Challenge (BraTS). Brisbane: The University of Queensland, 2015: 13-24.
19.	Ding X, Zhang X, Zhou Y, et al. Scaling up your kernels to 31×31: revisiting large kernel design in CNNs. arXiv preprint, 2022, arXiv: 1708.04231.
20.	Liu S, Chen T, Chen X, et al. More ConvNets in the 2020s: scaling up kernels beyond 51×51 using sparsity. arXiv preprint, 2023, arXiv: 2011.06641.
21.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Long Beach: Neural Inf Process Syst, 2017: 6000-6010.
22.	Ding X, Zhang X, Ma N, et al. RepVGG: making VGG-style ConvNets great again// IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville: IEEE, 2021: 13728-13737.
23.	Chen C, Dou Q, Jin Y, et al. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion//Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2020: 447-456.
24.	Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: Berkeley AI Res, 2016: 5967-5976.
25.	Chartsias A, Joyce T, Giuffrida M. V. , et al. Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans Med Imaging, 2018, 37(3): 803-814.
26.	Osman A F I, Tamam N M. Deep learning-based convolutional neural network for intra-modality brain MRI synthesis. Journal of Applied Clinical Medical Physics, 2022, 23(4): e13530.
27.	Nie D, Trullo R, Lian J, et al. Medical image synthesis with context-aware generative adversarial networks//The International Conference on Medical Image Computing and Computer Assisted Intervention. Cham: Springer, 2017: 417-425.

1. Iglesias J E, Konukoglu E, Zikic D, et al. Is synthesizing MRI contrast useful for inter-modality analysis?. Med Image Comput Comput Assist Interv, 2013, 16(Pt 1): 631-638.
2. Lustig M, Donoho D, Pauly J M. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 2007, 58(6): 1182-1195.
3. Zhou T, Fu H, Chen G, et al. Hi-Net: hybrid-fusion network for multi-modal MR image synthesis. IEEE Transactions on Medical Imaging, 2022, 39(9): 2772-2781.
4. Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks//2017 IEEE International Conference on Computer Vision(ICCV). Venice: University of Venice, 2017: 2242-2251.
5. Galbusera F, Bassani T, Casaroli G, et al. Generative models: an upcoming innovation in musculoskeletal radiology? A preliminary test in spine imaging. European Radiology Experimental, 2018, 2(1): 29-33.
6. 楼鑫杰. 基于深度学习与多模态融合的快速MRI重建研究. 浙江: 浙江工业大学, 2021.
7. Yang X, Lin Y, Wang Z, et al. Bi-modality medical image synthesis using semi-supervised sequential generative adversarial networks. IEEE Journal of Biomedical and Health Informatics, 2020, 24(3): 855-865.
8. Kim S, Jang H, Jang J, et al. Deep-learned short tau inversion recovery imaging using multi-contrast MR images. Magnetic Resonance in Medicine, 2020, 84(6): 2994-3008.
9. Torrado-Carvajal A, Herraiz J L, Alcain E, et al. Fast patch-based pseudo-CT synthesis from T1-weighted MR images for PET/MR attenuation correction in brain studies. The Journal of Nuclear Medicine, 2016, 57(1): 136-143.
10. Havaei M, Guizard N, Chapados N, et al. HeMIS: hetero-modal image segmentation //Medical Image Computing and Computer-Assisted Intervention (MICCAI). Athens: Imagia Inc, 2016: 469-477.
11. Fei Y, Zhan B, Hong M, et al. Deep learning-based multi-modal computing with feature disentanglement for MRI image synthesis. Medical Physics, 2021, 48(7): 3378-3789.
12. Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets//The International Conference on Neural Information Processing Systems. Montreal: NIPS, 2014: 2672-2680.
13. Baid U, Ghodasara S, Mohan S, et al. The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint, 2021, arXiv: 2107.02314.
14. Menze B H, Jakab A, Bauer S, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 2015, 34(10): 1993-2024.
15. Bakas S, Reyes M, Jakab A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint, 2021, arXiv: 1811.02629.
16. Feng C M, Yan Y, Liu C, et al. Exploring separable attention for multi-contrast MR image super-resolution. arXiv preprint, 2021, arXiv: 2109.01664.
17. 李英. 基于结构相似性和低秩稀疏的磁共振图像去噪算法研究. 计算机应用, 2017, 37(5): 1375-1379.
18. Dvorák P, Menze B. Structured prediction with convolutional neural networks for multimodal brain tumor segmentation// MICCAI Multimodal Brain Tumor Segmentation Challenge (BraTS). Brisbane: The University of Queensland, 2015: 13-24.
19. Ding X, Zhang X, Zhou Y, et al. Scaling up your kernels to 31×31: revisiting large kernel design in CNNs. arXiv preprint, 2022, arXiv: 1708.04231.
20. Liu S, Chen T, Chen X, et al. More ConvNets in the 2020s: scaling up kernels beyond 51×51 using sparsity. arXiv preprint, 2023, arXiv: 2011.06641.
21. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Long Beach: Neural Inf Process Syst, 2017: 6000-6010.
22. Ding X, Zhang X, Ma N, et al. RepVGG: making VGG-style ConvNets great again// IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville: IEEE, 2021: 13728-13737.
23. Chen C, Dou Q, Jin Y, et al. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion//Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2020: 447-456.
24. Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: Berkeley AI Res, 2016: 5967-5976.
25. Chartsias A, Joyce T, Giuffrida M. V. , et al. Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans Med Imaging, 2018, 37(3): 803-814.
26. Osman A F I, Tamam N M. Deep learning-based convolutional neural network for intra-modality brain MRI synthesis. Journal of Applied Clinical Medical Physics, 2022, 23(4): e13530.
27. Nie D, Trullo R, Lian J, et al. Medical image synthesis with context-aware generative adversarial networks//The International Conference on Medical Image Computing and Computer Assisted Intervention. Cham: Springer, 2017: 417-425.

Previous Article
Oral panorama reconstruction method based on pre-segmentation and Bezier function
Next Article
Non-local attention and multi-task learning based lung segmentation in chest X-ray

Journal of Biomedical Engineering

Deep learning method for magnetic resonance imaging fluid-attenuated inversion recovery image synthesis

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content