Research progress and challenges of deep learning in medical image registration_Journal of Biomedical Engineering

Authors：

ZOU Maoyang ^1,2 , YANG Hao ¹ , PAN Guanghui ¹ ,  ZHONG Yong ²

1. Chengdu University of Information Technology, Chengdu 610225, P.R.China;
2. Chengdu Institute of Computer Application, University of Chinese Academy of Sciences, Chengdu 610041, P.R.China;

Corresponding author：

Keywords：

medical image registration; deep learning; convolutional neural networks; fully convolutional networks

DOI：

10.7507/1001-5515.201810004

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

With the development of image-guided surgery and radiotherapy, the demand for medical image registration is stronger and the challenge is greater. In recent years, deep learning, especially deep convolution neural networks, has made excellent achievements in medical image processing, and its research in registration has developed rapidly. In this paper, the research progress of medical image registration based on deep learning at home and abroad is reviewed according to the category of technical methods, which include similarity measurement with an iterative optimization strategy, direct estimation of transform parameters, etc. Then, the challenge of deep learning in medical image registration is analyzed, and the possible solutions and open research are proposed.

Citation： ZOU Maoyang, YANG Hao, PAN Guanghui, ZHONG Yong. Research progress and challenges of deep learning in medical image registration. Journal of Biomedical Engineering, 2019, 36(4): 677-683. doi: 10.7507/1001-5515.201810004 Copy

1.	Shan S Y, Yan W, Guo X Q, et al. Unsupervised end-to-end learning for deformable medical image registration, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1711.08608v2.
2.	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks//25th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc, 2012: 1097-1105.
3.	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation //IEEE Conference on Computer Vision and Pattern Recognition, Boston: IEEE Computer Society, 2015: 3431-3440.
4.	Ren Shaoqing, He Kaiming, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks//Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal: MIT Press, 2015: 91-99.
5.	Litjens G, Kooi T, Bejnordi B E, et al. A survey on deep learning in medical image analysis. Med Image Anal, 2017, 42(9): 60-88.
6.	Lee J G, Jun S, Cho Y W, et al. Deep learning in medical imaging: general overview. Korean Journal of Radiology, 2017, 18(4): 570-584.
7.	Shen D, Wu G, Suk H I. Deep learning in medical image analysis. Annu Rev Biomed Eng, 2017, 19(1): 221-248.
8.	Greenspan H, Ginneken B V, Summers R M. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging, 2016, 35(5): 1153-1159.
9.	刘飞, 张俊然, 杨豪. 基于深度学习的医学图像识别研究进展. 中国生物医学工程学报, 2018, (1): 86-94.
10.	李渊, 骆志刚, 管乃洋, 等. 生物医学数据分析中的深度学习方法应用. 生物化学与生物物理进展, 2016, (5): 472-483.
11.	田娟秀, 刘国才, 谷珊珊, 等. 医学图像分析深度学习方法研究与挑战. 自动化学报, 2018, 44(3): 401-424.
12.	Alam F, Rahman S U, Ullah S, et al. Medical image registration in image guided surgery: issues, challenges and research opportunities. Biocybernetics and Biomedical Engineering, 2018, 38(1): 71-89.
13.	Esteva A, Kuprel B, Novoa R A, et al. Corrigendum: dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 546: 686.
14.	Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 2016, 316(22): 2402-2410.
15.	Fan J, Cao X, Yap P T, et al. BIRNet: brain image registration using dual-supervised fully convolutional networks, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1802.04692.
16.	Cao X H, Yang J H, Zhang J, et al. Deformable image registration based on similarity-steered CNN regression// International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 300-308.
17.	Chen F, Wu D, Liao H G. Registration of CT and ultrasound images of the spine with neural network and orientation code mutual information//International Conference on Medical Imaging and Virtual Reality, 2016: 292-301.
18.	Cheng X, Zhang L, Zheng Y F. Deep similarity learning for multimodal medical images. Comput Method Biomech Biomed Eng: Imaging Vis, 2018, 6(3): 248-252.
19.	Sokooti H, Vos B D, Berendsen F, et al. Nonrigid image registration using multi-scale 3D convolutional neural networks//International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 232-239.
20.	Eppenhof K A J, Pluim J P W. Supervised local error estimation for nonlinear image registration using convolutional neural networks//Society of Photo-Optical Instrumentation Engineers (SPIE). San Diego: SPIE Conference Series, 2017: 10133.
21.	Simonovsky M, Gutiérrez-Becker B, Mateus D, et al. A deep metric for multimodal registration//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2016). Athens: Springer International Publishing, 2016: 10-18.
22.	Liao R, Miao S, Tournemire P D, et al. An artificial agent for robust image registration// Thirty-First AAAI Conference on Artificial Intelligence. San Francisco: eprint arXiv: 1611.10336, 2016.
23.	Miao Shun, Wang Z J, Liao Rui. A CNN regression approach for real-time 2D/3D registration. IEEE Trans Med Imaging, 2016, 35(5): 1352-1363.
24.	Yang Xiao, Kwitt R, Styner M, et al. Quicksilver: fast predictive image registration - a deep learning approach. Neuroimage, 2017, 158: 378-396.
25.	Zheng Jiannan, Miao Shun, Wang Z J, et al. Pairwise domain adaptation module for CNN-based 2-D/3-D registration. Journal of Medical Imaging, 2018, 5(2): 021204.
26.	Uzunova H, Wilms M, Handels H, et al. Training CNNs for image registration from few samples with model-based data augmentation// International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 223-231.
27.	Wu Guorong, Kim M, Wang Qian, et al. Unsupervised deep feature learning for deformable registration of Mr brain images//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2013), 2013: 649-656.
28.	Marc-Michel Rohé, Datar M, Heimann T, et al. SVF-net: learning deformable image registration using shape matching//International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 266-274.
29.	Li H M, Fan Y. Non-rigid image registration using fully convolutional networks with deep self-supervision, arXiv.org e-Print Archive, 2017. https://arxiv.org/abs/1709.00799.
30.	Chen Z T, Fu Y W, Zhang Y D, et al. Semantic feature augmentation in few-shot learning, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1804.05298v3.
31.	Wang Yuxiong, Hebert M. Learning to learn: model regression networks for easy small sample learning// European Conference on Computer Vision (ECCV 2016), Amsterdam: Springer International Publishing, 2016: 616-634.
32.	Li Zhizhong, Hoiem D. Learning without forgetting// European Conference on Computer Vision (ECCV 2016), Amsterdam: Springer International Publishing, 2016: 614-629.
33.	Fu Yanwei, Sigal L. Semi-supervised vocabulary-informed learning//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 5337-5346.
34.	Zhu Junyan, Park T, Isola P, et al. Unpaired Image-to-Image translation using cycle-consistent adversarial networks//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 2242-2251.
35.	Mao Xudong, Li Qing, Xie Haoran, et al. Least squares generative adversarial networks//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 2813-2821.
36.	Durugkar I, Gemp I, Mahadevan S. Generative Multi-Adversarial Networks// 5th International Conference on Learning Representations (ICLR). Toulon, 2017.
37.	Dixit M, Kwitt R, Niethammer M, et al. AGA: attribute-guided augmentation//30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu: IEEE Computer Society, 2017: 3328-3336.
38.	Zou Maoyang, Zhong Yong. Transfer learning for classification of optical satellite image. Sensing and Imaging, 2018, 19(1): 6-19.
39.	Shin H C, Roth H R, Gao Mingchen, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
40.	Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks, arXiv.org e-Print Archive, 2015. https://arxiv.org/abs/1506.02025.
41.	de Vos B D, Berendsen F F, Viergever M A, et al. End-to-end unsupervised deformable image registration with a convolutional neural network// International Workshop on Deep Learning in Medical Image Analysis (DLMIA 2017, ML-CDS 2017). Springer, Cham. 2017, 10553: 204-212.
42.	Balakrishnan G, Zhao A, Sabuncu M R, et al. An unsupervised learning model for deformable medical image registration, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1802.02604.

1. Shan S Y, Yan W, Guo X Q, et al. Unsupervised end-to-end learning for deformable medical image registration, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1711.08608v2.
2. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks//25th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc, 2012: 1097-1105.
3. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation //IEEE Conference on Computer Vision and Pattern Recognition, Boston: IEEE Computer Society, 2015: 3431-3440.
4. Ren Shaoqing, He Kaiming, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks//Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal: MIT Press, 2015: 91-99.
5. Litjens G, Kooi T, Bejnordi B E, et al. A survey on deep learning in medical image analysis. Med Image Anal, 2017, 42(9): 60-88.
6. Lee J G, Jun S, Cho Y W, et al. Deep learning in medical imaging: general overview. Korean Journal of Radiology, 2017, 18(4): 570-584.
7. Shen D, Wu G, Suk H I. Deep learning in medical image analysis. Annu Rev Biomed Eng, 2017, 19(1): 221-248.
8. Greenspan H, Ginneken B V, Summers R M. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging, 2016, 35(5): 1153-1159.
9. 刘飞, 张俊然, 杨豪. 基于深度学习的医学图像识别研究进展. 中国生物医学工程学报, 2018, (1): 86-94.
10. 李渊, 骆志刚, 管乃洋, 等. 生物医学数据分析中的深度学习方法应用. 生物化学与生物物理进展, 2016, (5): 472-483.
11. 田娟秀, 刘国才, 谷珊珊, 等. 医学图像分析深度学习方法研究与挑战. 自动化学报, 2018, 44(3): 401-424.
12. Alam F, Rahman S U, Ullah S, et al. Medical image registration in image guided surgery: issues, challenges and research opportunities. Biocybernetics and Biomedical Engineering, 2018, 38(1): 71-89.
13. Esteva A, Kuprel B, Novoa R A, et al. Corrigendum: dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 546: 686.
14. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 2016, 316(22): 2402-2410.
15. Fan J, Cao X, Yap P T, et al. BIRNet: brain image registration using dual-supervised fully convolutional networks, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1802.04692.
16. Cao X H, Yang J H, Zhang J, et al. Deformable image registration based on similarity-steered CNN regression// International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 300-308.
17. Chen F, Wu D, Liao H G. Registration of CT and ultrasound images of the spine with neural network and orientation code mutual information//International Conference on Medical Imaging and Virtual Reality, 2016: 292-301.
18. Cheng X, Zhang L, Zheng Y F. Deep similarity learning for multimodal medical images. Comput Method Biomech Biomed Eng: Imaging Vis, 2018, 6(3): 248-252.
19. Sokooti H, Vos B D, Berendsen F, et al. Nonrigid image registration using multi-scale 3D convolutional neural networks//International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 232-239.
20. Eppenhof K A J, Pluim J P W. Supervised local error estimation for nonlinear image registration using convolutional neural networks//Society of Photo-Optical Instrumentation Engineers (SPIE). San Diego: SPIE Conference Series, 2017: 10133.
21. Simonovsky M, Gutiérrez-Becker B, Mateus D, et al. A deep metric for multimodal registration//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2016). Athens: Springer International Publishing, 2016: 10-18.
22. Liao R, Miao S, Tournemire P D, et al. An artificial agent for robust image registration// Thirty-First AAAI Conference on Artificial Intelligence. San Francisco: eprint arXiv: 1611.10336, 2016.
23. Miao Shun, Wang Z J, Liao Rui. A CNN regression approach for real-time 2D/3D registration. IEEE Trans Med Imaging, 2016, 35(5): 1352-1363.
24. Yang Xiao, Kwitt R, Styner M, et al. Quicksilver: fast predictive image registration - a deep learning approach. Neuroimage, 2017, 158: 378-396.
25. Zheng Jiannan, Miao Shun, Wang Z J, et al. Pairwise domain adaptation module for CNN-based 2-D/3-D registration. Journal of Medical Imaging, 2018, 5(2): 021204.
26. Uzunova H, Wilms M, Handels H, et al. Training CNNs for image registration from few samples with model-based data augmentation// International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 223-231.
27. Wu Guorong, Kim M, Wang Qian, et al. Unsupervised deep feature learning for deformable registration of Mr brain images//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2013), 2013: 649-656.
28. Marc-Michel Rohé, Datar M, Heimann T, et al. SVF-net: learning deformable image registration using shape matching//International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017). Quebec: Springer, Cham, 2017: 266-274.
29. Li H M, Fan Y. Non-rigid image registration using fully convolutional networks with deep self-supervision, arXiv.org e-Print Archive, 2017. https://arxiv.org/abs/1709.00799.
30. Chen Z T, Fu Y W, Zhang Y D, et al. Semantic feature augmentation in few-shot learning, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1804.05298v3.
31. Wang Yuxiong, Hebert M. Learning to learn: model regression networks for easy small sample learning// European Conference on Computer Vision (ECCV 2016), Amsterdam: Springer International Publishing, 2016: 616-634.
32. Li Zhizhong, Hoiem D. Learning without forgetting// European Conference on Computer Vision (ECCV 2016), Amsterdam: Springer International Publishing, 2016: 614-629.
33. Fu Yanwei, Sigal L. Semi-supervised vocabulary-informed learning//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 5337-5346.
34. Zhu Junyan, Park T, Isola P, et al. Unpaired Image-to-Image translation using cycle-consistent adversarial networks//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 2242-2251.
35. Mao Xudong, Li Qing, Xie Haoran, et al. Least squares generative adversarial networks//2017 IEEE International Conference on Computer Vision (ICCV), 2017: 2813-2821.
36. Durugkar I, Gemp I, Mahadevan S. Generative Multi-Adversarial Networks// 5th International Conference on Learning Representations (ICLR). Toulon, 2017.
37. Dixit M, Kwitt R, Niethammer M, et al. AGA: attribute-guided augmentation//30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu: IEEE Computer Society, 2017: 3328-3336.
38. Zou Maoyang, Zhong Yong. Transfer learning for classification of optical satellite image. Sensing and Imaging, 2018, 19(1): 6-19.
39. Shin H C, Roth H R, Gao Mingchen, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
40. Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks, arXiv.org e-Print Archive, 2015. https://arxiv.org/abs/1506.02025.
41. de Vos B D, Berendsen F F, Viergever M A, et al. End-to-end unsupervised deformable image registration with a convolutional neural network// International Workshop on Deep Learning in Medical Image Analysis (DLMIA 2017, ML-CDS 2017). Springer, Cham. 2017, 10553: 204-212.
42. Balakrishnan G, Zhao A, Sabuncu M R, et al. An unsupervised learning model for deformable medical image registration, arXiv.org e-Print Archive, 2018. https://arxiv.org/abs/1802.02604.

Journal of Biomedical Engineering

Research progress and challenges of deep learning in medical image registration

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content