CT and MRI fusion based on generative adversarial network and convolutional neural networks under image enhancement_Journal of Biomedical Engineering

Authors：

LIU Yunpeng ¹ ,  LI Jin ² , WANG Yu ¹ , CAI Wenli ³ , CHEN Fei ² , LIU Wenjie ¹ , MAO Xianhao ¹ , GAN Kaifeng ⁴ , WANG Renfang ² , SUN Dechao ⁵ , QIU Hong ² , LIU Bangquan ⁵

1. Information and Computing Science Major, School of International Exchange, Ningbo University of Technology, Ningbo, Zhejiang 315000, P. R. China;
2. Zhejiang Wanli University, Ningbo, Zhejiang 315000, P. R. China;
3. Radiology Imaging Laboratory, Harvard Medical School, Boston, Massachusetts 02114, USA;
4. Li Huili Hospital Affiliated to Ningbo University, Ningbo, Zhejiang 315000, P. R. China;
5. School of Digital Technology and Engineering, Ningbo University of Finance & Economics, Ningbo, Zhejiang 315000, P. R. China;

Corresponding author：

LI Jin, Email: 691920847@qq.com

Keywords：

Image enhancement; Image fusion; Generative adversarial network; Deep learning; Medical image

DOI：

10.7507/1001-5515.202209050

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Aiming at the problems of missing important features, inconspicuous details and unclear textures in the fusion of multimodal medical images, this paper proposes a method of computed tomography (CT) image and magnetic resonance imaging (MRI) image fusion using generative adversarial network (GAN) and convolutional neural network (CNN) under image enhancement. The generator aimed at high-frequency feature images and used double discriminators to target the fusion images after inverse transform; Then high-frequency feature images were fused by trained GAN model, and low-frequency feature images were fused by CNN pre-training model based on transfer learning. Experimental results showed that, compared with the current advanced fusion algorithm, the proposed method had more abundant texture details and clearer contour edge information in subjective representation. In the evaluation of objective indicators, Q^AB/F, information entropy (IE), spatial frequency (SF), structural similarity (SSIM), mutual information (MI) and visual information fidelity for fusion (VIFF) were 2.0%, 6.3%, 7.0%, 5.5%, 9.0% and 3.3% higher than the best test results, respectively. The fused image can be effectively applied to medical diagnosis to further improve the diagnostic efficiency.

Citation： LIU Yunpeng, LI Jin, WANG Yu, CAI Wenli, CHEN Fei, LIU Wenjie, MAO Xianhao, GAN Kaifeng, WANG Renfang, SUN Dechao, QIU Hong, LIU Bangquan. CT and MRI fusion based on generative adversarial network and convolutional neural networks under image enhancement. Journal of Biomedical Engineering, 2023, 40(2): 208-216. doi: 10.7507/1001-5515.202209050 Copy

1.	Liu Y, Chen X, Ward R K, et al. Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Process Lett, 2019, 26(3): 485-489.
2.	Yan H, Li Z. A multi-modal medical image fusion method in spatial domain// 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Chengdu: IEEE, 2019: 597-601.
3.	Li W, Jia L, Du J. Multi-modal sensor medical image fusion based on multiple salient features with guided image filter. IEEE Access, 2019, 7: 173019-173033.
4.	Yadav S P, Yadav S. Image fusion using hybrid methods in multimodality medical images. Med Biol Eng Comput, 2020, 58(4): 669-687.
5.	Zhang S, Li X, Zhu R, et al. Medical image fusion algorithm based on L0 gradient minimization for CT and MRI. Multimed Tools Appl, 2021, 80(14): 21135-21164.
6.	Irshad M T, Rehman H U. Gradient compass-based adaptive multimodal medical image fusion. IEEE Access, 2021, 9: 22662-22670.
7.	Zhu Z, Zheng M, Qi G, et al. A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain. IEEE Access, 2019, 7: 20811-20824.
8.	Gai D, Shen X, Cheng H, et al. Medical image fusion via PCNN based on edge preservation and improved sparse representation in NSST domain. IEEE Access, 2019, 7: 85413-85429.
9.	Singh S, Anand R S. Multimodal medical image sensor fusion model using sparse K-SVD dictionary learning in nonsubsampled shearlet domain. IEEE Trans Instrum Meas, 2019, 69(2): 593-607.
10.	Meng L, Guo X, Li H. MRI/CT fusion based on latent low rank representation and gradient transfer. Biomed Signal Process Control, 2019, 53: 101536.
11.	Ullah H, Ullah B, Wu L, et al. Multi-modality medical images fusion based on local-features fuzzy sets and novel sum-modified-Laplacian in non-subsampled shearlet transform domain. Biomed Signal Process Control, 2020, 57: 101724.
12.	Sayadi M, Ghassemian H, Naimi R, et al. A new composite multimodality image fusion method based on shearlet transform and retina inspired model// 2020 International Conference on Machine Vision and Image Processing (MVIP). Iran: IEEE, 2020: 1-5.
13.	Panigrahy C, Seal A, Mahato N K. MRI and SPECT image fusion using a weighted parameter adaptive dual channel PCNN. IEEE Signal Process Lett, 2020, 27: 690-694.
14.	Zhu R, Li X, Zhang X, et al. MRI and CT medical image fusion based on synchronized-anisotropic diffusion model. IEEE Access, 2020, 8: 91336-91350.
15.	Goyal B, Dogra A, Khoond R, et al. An efficient medical assistive diagnostic algorithm for visualisation of structural and tissue details in CT and MRI fusion. Cognit Comput, 2021, 13(6): 1471-1483.
16.	Zhang L, Zhang Y, Ma S, et al. CT and MRI image fusion algorithm based on hybrid ℓ0ℓ1 layer decomposing and two-dimensional variation transform. Biomed Signal Process Control, 2021, 70: 103024.
17.	Zhu R, Li X, Zhang X, et al. HID: the hybrid image decomposition model for MRI and CT fusion. IEEE J Biomed Health Inform, 2021, 26(2): 727-739.
18.	Faragallah O S, Muhammed A N, Taha T S, et al. PCA based SVD fusion for MRI and CT medical images. J Intell Fuzzy Syst, 2021, 41(2): 4021-4033.
19.	Polinati S, Bavirisetti D P, Rajesh K N, et al. The fusion of MRI and CT medical images using variational mode decomposition. Appl Sci, 2021, 11(22): 10975.
20.	Nian Z, Jung C. CNN-based multi-focus image fusion with light field data// 2019 IEEE International Conference on Image Processing (ICIP). Taipei: IEEE, 2019: 1044-1048.
21.	Liang X, Hu P, Zhang L, et al. MCFNet: multi-layer concatenation fusion network for medical images fusion. IEEE Sens J, 2019, 19(16): 7107-7119.
22.	Maneesha P, Singh T, Nayar R, et al. Multi modal medical image fusion using convolution neural network// 2019 Third International Conference on Inventive Systems and Control (ICISC). Coimbatore: IEEE, 2019: 351-357.
23.	Zhang Y, Liu Y, Sun P, et al. IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion, 2020, 54: 99-118.
24.	Wang K, Zheng M, Wei H, et al. Multi-modality medical image fusion using convolutional neural network and contrast pyramid. Sensors, 2020, 20(8): 2169.
25.	Ma J, Yu W, Liang P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion. Inform Fusion, 2019, 48: 11-26.
26.	Xu H, Liang P, Yu W, et al. Learning a generative model for fusing infrared and visible images via conditional generative adversarial network with dual discriminators// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI). Macao: ACM, 2019: 3954-3960.
27.	Ma J, Xu H, Jiang J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process, 2020, 29: 4980-4995.
28.	Isola P, Zhu J Y, Zhou T H, et al. Image-to-image translation with conditional adversarial networks// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5967-5976.
29.	Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
30.	Oktay O, Schlemper J, Folgoc L L, et al. Attention U-Net: Learning where to look for the pancreas. arXiv, 2018: 1804.03999.

1. Liu Y, Chen X, Ward R K, et al. Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Process Lett, 2019, 26(3): 485-489.
2. Yan H, Li Z. A multi-modal medical image fusion method in spatial domain// 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Chengdu: IEEE, 2019: 597-601.
3. Li W, Jia L, Du J. Multi-modal sensor medical image fusion based on multiple salient features with guided image filter. IEEE Access, 2019, 7: 173019-173033.
4. Yadav S P, Yadav S. Image fusion using hybrid methods in multimodality medical images. Med Biol Eng Comput, 2020, 58(4): 669-687.
5. Zhang S, Li X, Zhu R, et al. Medical image fusion algorithm based on L0 gradient minimization for CT and MRI. Multimed Tools Appl, 2021, 80(14): 21135-21164.
6. Irshad M T, Rehman H U. Gradient compass-based adaptive multimodal medical image fusion. IEEE Access, 2021, 9: 22662-22670.
7. Zhu Z, Zheng M, Qi G, et al. A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain. IEEE Access, 2019, 7: 20811-20824.
8. Gai D, Shen X, Cheng H, et al. Medical image fusion via PCNN based on edge preservation and improved sparse representation in NSST domain. IEEE Access, 2019, 7: 85413-85429.
9. Singh S, Anand R S. Multimodal medical image sensor fusion model using sparse K-SVD dictionary learning in nonsubsampled shearlet domain. IEEE Trans Instrum Meas, 2019, 69(2): 593-607.
10. Meng L, Guo X, Li H. MRI/CT fusion based on latent low rank representation and gradient transfer. Biomed Signal Process Control, 2019, 53: 101536.
11. Ullah H, Ullah B, Wu L, et al. Multi-modality medical images fusion based on local-features fuzzy sets and novel sum-modified-Laplacian in non-subsampled shearlet transform domain. Biomed Signal Process Control, 2020, 57: 101724.
12. Sayadi M, Ghassemian H, Naimi R, et al. A new composite multimodality image fusion method based on shearlet transform and retina inspired model// 2020 International Conference on Machine Vision and Image Processing (MVIP). Iran: IEEE, 2020: 1-5.
13. Panigrahy C, Seal A, Mahato N K. MRI and SPECT image fusion using a weighted parameter adaptive dual channel PCNN. IEEE Signal Process Lett, 2020, 27: 690-694.
14. Zhu R, Li X, Zhang X, et al. MRI and CT medical image fusion based on synchronized-anisotropic diffusion model. IEEE Access, 2020, 8: 91336-91350.
15. Goyal B, Dogra A, Khoond R, et al. An efficient medical assistive diagnostic algorithm for visualisation of structural and tissue details in CT and MRI fusion. Cognit Comput, 2021, 13(6): 1471-1483.
16. Zhang L, Zhang Y, Ma S, et al. CT and MRI image fusion algorithm based on hybrid ℓ0ℓ1 layer decomposing and two-dimensional variation transform. Biomed Signal Process Control, 2021, 70: 103024.
17. Zhu R, Li X, Zhang X, et al. HID: the hybrid image decomposition model for MRI and CT fusion. IEEE J Biomed Health Inform, 2021, 26(2): 727-739.
18. Faragallah O S, Muhammed A N, Taha T S, et al. PCA based SVD fusion for MRI and CT medical images. J Intell Fuzzy Syst, 2021, 41(2): 4021-4033.
19. Polinati S, Bavirisetti D P, Rajesh K N, et al. The fusion of MRI and CT medical images using variational mode decomposition. Appl Sci, 2021, 11(22): 10975.
20. Nian Z, Jung C. CNN-based multi-focus image fusion with light field data// 2019 IEEE International Conference on Image Processing (ICIP). Taipei: IEEE, 2019: 1044-1048.
21. Liang X, Hu P, Zhang L, et al. MCFNet: multi-layer concatenation fusion network for medical images fusion. IEEE Sens J, 2019, 19(16): 7107-7119.
22. Maneesha P, Singh T, Nayar R, et al. Multi modal medical image fusion using convolution neural network// 2019 Third International Conference on Inventive Systems and Control (ICISC). Coimbatore: IEEE, 2019: 351-357.
23. Zhang Y, Liu Y, Sun P, et al. IFCNN: a general image fusion framework based on convolutional neural network. Information Fusion, 2020, 54: 99-118.
24. Wang K, Zheng M, Wei H, et al. Multi-modality medical image fusion using convolutional neural network and contrast pyramid. Sensors, 2020, 20(8): 2169.
25. Ma J, Yu W, Liang P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion. Inform Fusion, 2019, 48: 11-26.
26. Xu H, Liang P, Yu W, et al. Learning a generative model for fusing infrared and visible images via conditional generative adversarial network with dual discriminators// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI). Macao: ACM, 2019: 3954-3960.
27. Ma J, Xu H, Jiang J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process, 2020, 29: 4980-4995.
28. Isola P, Zhu J Y, Zhou T H, et al. Image-to-image translation with conditional adversarial networks// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5967-5976.
29. Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
30. Oktay O, Schlemper J, Folgoc L L, et al. Attention U-Net: Learning where to look for the pancreas. arXiv, 2018: 1804.03999.

Journal of Biomedical Engineering

CT and MRI fusion based on generative adversarial network and convolutional neural networks under image enhancement

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content