A multimodal medical image contrastive learning algorithm with domain adaptive denormalization_Journal of Biomedical Engineering

Authors：

WEN Han ^1,2 , ZHAO Ying ³ , CAI Xiuding ^1,2 ,  LIU Ailian ^3,4 , YAO Yu ^1,2 , FU Zhongliang ^1,2

1. Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu 610213, P. R. China;
2. University of Chinese Academy of Sciences, Beijing 100049, P. R. China;
3. The First Affiliated Hospital of Dalian Medical University, Dalian, Liaoning 116011, P. R. China;
4. Dalian Medical Imaging Artificial Intelligence Engineering Technology Research Center, Dalian, Liaoning 116000, P. R. China;

Corresponding author：

LIU Ailian, Email: liuailian@dmu.edu.cn

Keywords：

Self-supervised learning; Multimodal medical image; Disease diagnosis; Domain adaptive denormalization

DOI：

10.7507/1001-5515.202302050

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Recently, deep learning has achieved impressive results in medical image tasks. However, this method usually requires large-scale annotated data, and medical images are expensive to annotate, so it is a challenge to learn efficiently from the limited annotated data. Currently, the two commonly used methods are transfer learning and self-supervised learning. However, these two methods have been little studied in multimodal medical images, so this study proposes a contrastive learning method for multimodal medical images. The method takes images of different modalities of the same patient as positive samples, which effectively increases the number of positive samples in the training process and helps the model to fully learn the similarities and differences of lesions on images of different modalities, thus improving the model's understanding of medical images and diagnostic accuracy. The commonly used data augmentation methods are not suitable for multimodal images, so this paper proposes a domain adaptive denormalization method to transform the source domain images with the help of statistical information of the target domain. In this study, the method is validated with two different multimodal medical image classification tasks: in the microvascular infiltration recognition task, the method achieves an accuracy of (74.79 ± 0.74)% and an F1 score of (78.37 ± 1.94)%, which are improved as compared with other conventional learning methods; for the brain tumor pathology grading task, the method also achieves significant improvements. The results show that the method achieves good results on multimodal medical images and can provide a reference solution for pre-training multimodal medical images.

Citation： WEN Han, ZHAO Ying, CAI Xiuding, LIU Ailian, YAO Yu, FU Zhongliang. A multimodal medical image contrastive learning algorithm with domain adaptive denormalization. Journal of Biomedical Engineering, 2023, 40(3): 482-491. doi: 10.7507/1001-5515.202302050 Copy

1.	Litjens G, Kooi T, Bejnordi B E, et al. A survey on deep learning in medical image analysis. Medical image analysis, 2017, 42: 60-88.
2.	Maghdid H S, Asaad A T, Ghafoor K Z, et al. Diagnosing COVID-19 pneumonia from X-ray and CT images using deep learning and transfer learning algorithms. arXiv preprint, 2021, arXiv: 2004.00038.
3.	Yang Y, Li X, Wang P, et al. Multi-Source transfer learning via ensemble approach for initial diagnosis of Alzheimer’s disease. IEEE Journal of Translational Engineering in Health and Medicine, 2020, 8: 1400310.
4.	Patrini I, Ruperti M, Moccia S, et al. Transfer learning for informative-frame selection in laryngoscopic videos through learned features. Medical & Biological Engineering & Computing, 2020, 58(6): 1225-1238.
5.	Zhu J, Li Y, Hu Y, et al. Rubik’s cube+: a self-supervised feature learning framework for 3D medical image analysis. Medical Image Analysis, 2020, 64: 101746.
6.	Li H, Xue F F, Chaitanya K, et al. Imbalance-aware self-supervised learning for 3D radiomic representations//International Conference on Medical Image Computing and Computer-Assisted Intervention. Switzerland: Springer, 2021: 36-46.
7.	Raghu M, Zhang C, Kleinberg J, et al. Transfusion: understanding transfer learning for medical imaging//International Conference on Neural Information Processing Systems, Vancouver: MIT Press, 2019: 3347-3357.
8.	Bai W, Chen C, Tarroni G, et al. Self-supervised learning for cardiac MR image segmentation by anatomical position prediction//Medical Image Computing and Computer Assisted Intervention(MICCAI 2019), Shenzhen: Springer, 2019: 541-549.
9.	Hervella Á S, Rouco J, Novo J, et al. Self-supervised multimodal reconstruction pre-training for retinal computer-aided diagnosis. Expert Systems with Applications, 2021, 185: 115598.
10.	He Xuehai, Yang Xingyi, Zhang Shuanghang, et al. Sample-efficient deep learning for COVID-19 diagnosis based on CT scans. arXiv preprint, 2020. DOI: 10.1101/2020.04.13.20063941.
11.	Li Yi, Zhao Junli, Lv Zhihan, et al. Medical image fusion method by deep learning. International Journal of Cognitive Computing in Engineering, 2021, 2: 21-29.
12.	Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations//International Conference on Machine Learning, Vienna: ACM, 2020: 1597-1607.
13.	He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle: IEEE, 2020: 9729-9738.
14.	Grill J B, Strub F, Altché F, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020, 33: 21271-21284.
15.	Chen Xinlei, He Kaiming. Exploring simple siamese representation learning//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville: IEEE, 2021: 15745-15753.
16.	Chaitanya K, Erdil E, Karani N, et al. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 2020, 33: 12546-12558.
17.	Sowrirajan H, Yang J, Ng A Y, et al. Moco pretraining improves representation and transferability of chest X-ray models// Medical Imaging with Deep Learning, Lubeck: MIDL, 2021: 728-744.
18.	Irvin J, Rajpurkar P, Ko M, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison//AAAI Conference on Artificial Intelligence. 2019, 33(1): 590-597.
19.	Azizi S, Mustafa B, Ryan F, et al. Big self-supervised models advance medical image classification//IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 3478-3488.
20.	Windsor R, Jamaludin A, Kadir T, et al. Self-supervised multi-modal alignment for whole body medical imaging//Medical Image Computing and Computer Assisted Intervention(MICCAI) 2021, Strasbourg: Springer, 2021: 90-101.
21.	Taleb A, Lippert C, Klein T, et al. Multimodal self-supervised learning for medical image analysis//Information Processing in Medical Imaging, arXiv preprint, 2020. arXiv: 1912.05396.
22.	Zhang Hongyi, Cisse M, Dauphin Y N, et al. Mixup: Beyond empirical risk minimization//International Conference on Learning Representations, arXiv preprint, 2018. arXiv: 1710.09412.
23.	Yun S, Han D, Oh S J, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features//IEEE/CVF International Conference on Computer Vision, Seoul: IEEE, 2019: 6023-6032.
24.	Zhang X, Zhou X, Lin M, et al. Shufflenet: an extremely efficient convolutional neural network for mobile devices//IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake: IEEE, 2018: 6848-6856.
25.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition//IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2016: 770-778.
26.	Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows//IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 10012-10022.
27.	He K, Zhang X, Ren S, et al. Delving deep into rectifiers: surpassing human-level performance on imagenet classification//IEEE International Conference on Computer Vision, Santiago: IEEE, 2015: 1026-1034.
28.	Luo Liangchen, Xiong Yuanhao, Liu Yan, et al. Adaptive gradient methods with dynamic bound of learning rate//International Conference on Learning Representations, arXiv preprint, 2019, arXiv: 1902.09843.

1. Litjens G, Kooi T, Bejnordi B E, et al. A survey on deep learning in medical image analysis. Medical image analysis, 2017, 42: 60-88.
2. Maghdid H S, Asaad A T, Ghafoor K Z, et al. Diagnosing COVID-19 pneumonia from X-ray and CT images using deep learning and transfer learning algorithms. arXiv preprint, 2021, arXiv: 2004.00038.
3. Yang Y, Li X, Wang P, et al. Multi-Source transfer learning via ensemble approach for initial diagnosis of Alzheimer’s disease. IEEE Journal of Translational Engineering in Health and Medicine, 2020, 8: 1400310.
4. Patrini I, Ruperti M, Moccia S, et al. Transfer learning for informative-frame selection in laryngoscopic videos through learned features. Medical & Biological Engineering & Computing, 2020, 58(6): 1225-1238.
5. Zhu J, Li Y, Hu Y, et al. Rubik’s cube+: a self-supervised feature learning framework for 3D medical image analysis. Medical Image Analysis, 2020, 64: 101746.
6. Li H, Xue F F, Chaitanya K, et al. Imbalance-aware self-supervised learning for 3D radiomic representations//International Conference on Medical Image Computing and Computer-Assisted Intervention. Switzerland: Springer, 2021: 36-46.
7. Raghu M, Zhang C, Kleinberg J, et al. Transfusion: understanding transfer learning for medical imaging//International Conference on Neural Information Processing Systems, Vancouver: MIT Press, 2019: 3347-3357.
8. Bai W, Chen C, Tarroni G, et al. Self-supervised learning for cardiac MR image segmentation by anatomical position prediction//Medical Image Computing and Computer Assisted Intervention(MICCAI 2019), Shenzhen: Springer, 2019: 541-549.
9. Hervella Á S, Rouco J, Novo J, et al. Self-supervised multimodal reconstruction pre-training for retinal computer-aided diagnosis. Expert Systems with Applications, 2021, 185: 115598.
10. He Xuehai, Yang Xingyi, Zhang Shuanghang, et al. Sample-efficient deep learning for COVID-19 diagnosis based on CT scans. arXiv preprint, 2020. DOI: 10.1101/2020.04.13.20063941.
11. Li Yi, Zhao Junli, Lv Zhihan, et al. Medical image fusion method by deep learning. International Journal of Cognitive Computing in Engineering, 2021, 2: 21-29.
12. Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations//International Conference on Machine Learning, Vienna: ACM, 2020: 1597-1607.
13. He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle: IEEE, 2020: 9729-9738.
14. Grill J B, Strub F, Altché F, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020, 33: 21271-21284.
15. Chen Xinlei, He Kaiming. Exploring simple siamese representation learning//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville: IEEE, 2021: 15745-15753.
16. Chaitanya K, Erdil E, Karani N, et al. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 2020, 33: 12546-12558.
17. Sowrirajan H, Yang J, Ng A Y, et al. Moco pretraining improves representation and transferability of chest X-ray models// Medical Imaging with Deep Learning, Lubeck: MIDL, 2021: 728-744.
18. Irvin J, Rajpurkar P, Ko M, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison//AAAI Conference on Artificial Intelligence. 2019, 33(1): 590-597.
19. Azizi S, Mustafa B, Ryan F, et al. Big self-supervised models advance medical image classification//IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 3478-3488.
20. Windsor R, Jamaludin A, Kadir T, et al. Self-supervised multi-modal alignment for whole body medical imaging//Medical Image Computing and Computer Assisted Intervention(MICCAI) 2021, Strasbourg: Springer, 2021: 90-101.
21. Taleb A, Lippert C, Klein T, et al. Multimodal self-supervised learning for medical image analysis//Information Processing in Medical Imaging, arXiv preprint, 2020. arXiv: 1912.05396.
22. Zhang Hongyi, Cisse M, Dauphin Y N, et al. Mixup: Beyond empirical risk minimization//International Conference on Learning Representations, arXiv preprint, 2018. arXiv: 1710.09412.
23. Yun S, Han D, Oh S J, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features//IEEE/CVF International Conference on Computer Vision, Seoul: IEEE, 2019: 6023-6032.
24. Zhang X, Zhou X, Lin M, et al. Shufflenet: an extremely efficient convolutional neural network for mobile devices//IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake: IEEE, 2018: 6848-6856.
25. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition//IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2016: 770-778.
26. Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows//IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 10012-10022.
27. He K, Zhang X, Ren S, et al. Delving deep into rectifiers: surpassing human-level performance on imagenet classification//IEEE International Conference on Computer Vision, Santiago: IEEE, 2015: 1026-1034.
28. Luo Liangchen, Xiong Yuanhao, Liu Yan, et al. Adaptive gradient methods with dynamic bound of learning rate//International Conference on Learning Representations, arXiv preprint, 2019, arXiv: 1902.09843.

Journal of Biomedical Engineering

A multimodal medical image contrastive learning algorithm with domain adaptive denormalization

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content