Cascaded multi-level medical image registration method based on transformer_Journal of Biomedical Engineering

Authors：

PAN Yingjie ¹ ,  CHENG Yuanzhi ^1,2 , LIU Hao ¹ , SHI Cao ¹

1. School of Information Science and Technology, QingDao University of Science and Technology, Qingdao, Shandong 266000, P. R. China;
2. School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, P. R. China;

Corresponding author：

CHENG Yuanzhi, Email: yzcheng@hitwh.edu.cn

Keywords：

Medical imaging; Multi-level registration; Difficult deformation perception; Cascading network; Self-attention mechanism

DOI：

10.7507/1001-5515.202204011

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

In deep learning-based image registration, the deformable region with complex anatomical structures is an important factor affecting the accuracy of network registration. However, it is difficult for existing methods to pay attention to complex anatomical regions of images. At the same time, the receptive field of the convolutional neural network is limited by the size of its convolution kernel, and it is difficult to learn the relationship between the voxels with far spatial location, making it difficult to deal with the large region deformation problem. Aiming at the above two problems, this paper proposes a cascaded multi-level registration network model based on transformer, and equipped it with a difficult deformable region perceptron based on mean square error. The difficult deformation perceptron uses sliding window and floating window techniques to retrieve the registered images, obtain the difficult deformation coefficient of each voxel, and identify the regions with the worst registration effect. In this study, the cascaded multi-level registration network model adopts the difficult deformation perceptron for hierarchical connection, and the self-attention mechanism is used to extract global features in the basic registration network to optimize the registration results of different scales. The experimental results show that the method proposed in this paper can perform progressive registration of complex deformation regions, thereby optimizing the registration results of brain medical images, which has a good auxiliary effect on the clinical diagnosis of doctors.

Citation： PAN Yingjie, CHENG Yuanzhi, LIU Hao, SHI Cao. Cascaded multi-level medical image registration method based on transformer. Journal of Biomedical Engineering, 2022, 39(5): 876-886. doi: 10.7507/1001-5515.202204011 Copy

1.	Haskins G, Kruger U, Yan Pingkun. Deep learning in medical image registration: a survey. arXiv: 1903.02026, 2020. https://doi.org/10.48550/arXiv.1903.02026.
2.	Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks//the 28th International Conference on Neural Information Processing Systems-Volume 2 (NIPS), 2015. https://doi.org/10.48550/arXiv.1506.02025.
3.	Balakrishnan G, Zhao A, Sabuncu M R, et al. An unsupervised learning model for deformable medical image registration//Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR). 2018. https://doi.org/10.48550/arXiv.1802.02604.
4.	Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation//International Conference on Medical image computing and computer-assisted intervention(MICCAI), Cham: Springer, 2015: 234-241.
5.	Zhao Shengyu, Dong Yue, Chang E I, et al. Recursive cascaded networks for unsupervised medical image registration//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV). 2019: 10600-10610. https://doi.org/10.48550/arXiv.1907.12353.
6.	Kim B, Kim D H, Park S H, et al. CycleMorph: cycle consistent unsupervised deformable image registration. Med Image Anal, 2021, 71: 102036.
7.	Huang Y, Ahmad S, Fan J, et al. Difficulty-aware hierarchical convolutional neural networks for deformable registration of brain MR images. Med Image Anal, 2021, 67: 101817.
8.	Li Shaohua, Sui Xiuchao, Luo Xiangde, et al. Medical image segmentation using squeeze-and-expansion transformers. arXiv: 2105.09511, 2021. https://doi.org/10.48550/arXiv.2105.09511.
9.	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv: 2010.11929, 2020. https://doi.org/10.48550/arXiv.2010.11929.
10.	Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows//The IEEE/CVF International Conference on Computer Vision(ICCV). 2021: 10012-10022.
11.	Chen Junyu, He Yufan, Frey E C, et al. ViT-V-Net: vision transformer for unsupervised volumetric medical image registration. arXiv: 2104.06468, 2021. https://doi.org/10.48550/arXiv.2104.06468.
12.	Milletari F, Navab N, Ahmadi S A. V-net: fully convolutional neural networks for volumetric medical image segmentation//2016 fourth international conference on 3D vision (3DV). IEEE, 2016: 565-571.
13.	Chen Junyu, Du Yong, He Yufan, et al. TransMorph: Transformer for unsupervised medical image registration. arXiv: 2111.10480, 2021. https://doi.org/10.48550/arXiv.2111.10480.
14.	Vercauteren T, Pennec X, Perchant A, et al. Diffeomorphic demons: efficient non-parametric image registration. Neuroimage, 2009, 45(1 Suppl): S61-S72.
15.	Zhou H Y, Guo J, Zhang Y, et al. nnFormer: interleaved transformer for volumetric segmentation. arXiv: 2109.03201, 2021. https://doi.org/10.48550/arXiv.2109.03201.
16.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach: NIPS, 2017: 6000-6010.
17.	Zeiler M D, Taylor G W, Fergus R. Adaptive deconvolutional networks for mid and high level feature learning//2011 International Conference on Computer Vision (ICCV). Barcelona: IEEE, 2011: 12491108.
18.	Allen D M. Mean square error of prediction as a criterion for selecting variables. Technometrics, 1971, 13(3): 469-475.
19.	Shattuck DW, Mirza M, Adisetiyo V, et al. Construction of a 3D probabilistic atlas of human cortical structures. Neuroimage, 2008, 39(3): 1064-1080.
20.	Marcus DS, Wang TH, Parker J, et al. Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J Cogn Neurosci, 2007, 19(9): 1498-1507.
21.	Fischl B. FreeSurfer. NeuroImage, 2012, 62(2): 774-781.
22.	Avants B B, Epstein C L, Grossman M, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal, 2008, 12(1): 26-41.
23.	Klein A, Andersson J, Ardekani B A, et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage, 2009, 46(3): 786-802.
24.	Avants B B, Tustison N J, Song G, et al. A reproducible evaluation of ANTs similarity metric performance in brain image registration. NeuroImage, 2011, 54(3): 2033-2044.
25.	Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library// the 33rd International Conference on Neural Information Processing Systems (NeurIPS 2019), 2019. https://doi.org/10.48550/arXiv.1912.01703.
26.	Dice L R. Measures of the amount of ecologic association between species. Ecology, 1945, 26(3): 297-302.
27.	Dacorogna B, Moser J. On a partial differential equation involving the jacobian determinant. Annales de l'Institut Henri Poincaré C, Analyse non linéaire, 1990, 7(1): 1-26.
28.	Wang Shiqi, Rehman A, Wang Zhou, et al. SSIM-motivated rate-distortion optimization for video coding. IEEE Transactions on Circuits and Systems for Video Technology, 2011, 22(4): 516-529.
29.	Viola P, Wells III W M. Alignment by maximization of mutual information//IEEE International Conference on Computer Vision, 1995: 16-23. DOI: 10.1109/ICCV.1995.466930.

1. Haskins G, Kruger U, Yan Pingkun. Deep learning in medical image registration: a survey. arXiv: 1903.02026, 2020. https://doi.org/10.48550/arXiv.1903.02026.
2. Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks//the 28th International Conference on Neural Information Processing Systems-Volume 2 (NIPS), 2015. https://doi.org/10.48550/arXiv.1506.02025.
3. Balakrishnan G, Zhao A, Sabuncu M R, et al. An unsupervised learning model for deformable medical image registration//Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR). 2018. https://doi.org/10.48550/arXiv.1802.02604.
4. Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation//International Conference on Medical image computing and computer-assisted intervention(MICCAI), Cham: Springer, 2015: 234-241.
5. Zhao Shengyu, Dong Yue, Chang E I, et al. Recursive cascaded networks for unsupervised medical image registration//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV). 2019: 10600-10610. https://doi.org/10.48550/arXiv.1907.12353.
6. Kim B, Kim D H, Park S H, et al. CycleMorph: cycle consistent unsupervised deformable image registration. Med Image Anal, 2021, 71: 102036.
7. Huang Y, Ahmad S, Fan J, et al. Difficulty-aware hierarchical convolutional neural networks for deformable registration of brain MR images. Med Image Anal, 2021, 67: 101817.
8. Li Shaohua, Sui Xiuchao, Luo Xiangde, et al. Medical image segmentation using squeeze-and-expansion transformers. arXiv: 2105.09511, 2021. https://doi.org/10.48550/arXiv.2105.09511.
9. Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv: 2010.11929, 2020. https://doi.org/10.48550/arXiv.2010.11929.
10. Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows//The IEEE/CVF International Conference on Computer Vision(ICCV). 2021: 10012-10022.
11. Chen Junyu, He Yufan, Frey E C, et al. ViT-V-Net: vision transformer for unsupervised volumetric medical image registration. arXiv: 2104.06468, 2021. https://doi.org/10.48550/arXiv.2104.06468.
12. Milletari F, Navab N, Ahmadi S A. V-net: fully convolutional neural networks for volumetric medical image segmentation//2016 fourth international conference on 3D vision (3DV). IEEE, 2016: 565-571.
13. Chen Junyu, Du Yong, He Yufan, et al. TransMorph: Transformer for unsupervised medical image registration. arXiv: 2111.10480, 2021. https://doi.org/10.48550/arXiv.2111.10480.
14. Vercauteren T, Pennec X, Perchant A, et al. Diffeomorphic demons: efficient non-parametric image registration. Neuroimage, 2009, 45(1 Suppl): S61-S72.
15. Zhou H Y, Guo J, Zhang Y, et al. nnFormer: interleaved transformer for volumetric segmentation. arXiv: 2109.03201, 2021. https://doi.org/10.48550/arXiv.2109.03201.
16. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach: NIPS, 2017: 6000-6010.
17. Zeiler M D, Taylor G W, Fergus R. Adaptive deconvolutional networks for mid and high level feature learning//2011 International Conference on Computer Vision (ICCV). Barcelona: IEEE, 2011: 12491108.
18. Allen D M. Mean square error of prediction as a criterion for selecting variables. Technometrics, 1971, 13(3): 469-475.
19. Shattuck DW, Mirza M, Adisetiyo V, et al. Construction of a 3D probabilistic atlas of human cortical structures. Neuroimage, 2008, 39(3): 1064-1080.
20. Marcus DS, Wang TH, Parker J, et al. Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J Cogn Neurosci, 2007, 19(9): 1498-1507.
21. Fischl B. FreeSurfer. NeuroImage, 2012, 62(2): 774-781.
22. Avants B B, Epstein C L, Grossman M, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal, 2008, 12(1): 26-41.
23. Klein A, Andersson J, Ardekani B A, et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage, 2009, 46(3): 786-802.
24. Avants B B, Tustison N J, Song G, et al. A reproducible evaluation of ANTs similarity metric performance in brain image registration. NeuroImage, 2011, 54(3): 2033-2044.
25. Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library// the 33rd International Conference on Neural Information Processing Systems (NeurIPS 2019), 2019. https://doi.org/10.48550/arXiv.1912.01703.
26. Dice L R. Measures of the amount of ecologic association between species. Ecology, 1945, 26(3): 297-302.
27. Dacorogna B, Moser J. On a partial differential equation involving the jacobian determinant. Annales de l'Institut Henri Poincaré C, Analyse non linéaire, 1990, 7(1): 1-26.
28. Wang Shiqi, Rehman A, Wang Zhou, et al. SSIM-motivated rate-distortion optimization for video coding. IEEE Transactions on Circuits and Systems for Video Technology, 2011, 22(4): 516-529.
29. Viola P, Wells III W M. Alignment by maximization of mutual information//IEEE International Conference on Computer Vision, 1995: 16-23. DOI: 10.1109/ICCV.1995.466930.

Journal of Biomedical Engineering

Cascaded multi-level medical image registration method based on transformer

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content