Hepatocellular carcinoma segmentation and pathological differentiation degree prediction method based on multi-task learning_Journal of Biomedical Engineering

Authors：

WEN Han ^1,2 , ZHAO Ying ³ , YANG Yong ^1,2 , WANG Hongkai ^4,5 ,  LIU Ailian ^3,5 , YAO Yu ^1,2 , FU Zhongliang ^1,2

1. Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu 610213, P. R. China;
2. University of Chinese Academy of Sciences, Beijing 100049, P. R. China;
3. The First Affiliated Hospital of Dalian Medical University, Dalian, Liaoning 116011, P. R. China;
4. School of Biomedical Engineering, Dalian University of Technology, Dalian, Liaoning 116024, P. R. China;
5. Dalian Medical Imaging Artificial Intelligence Engineering Technology Research Center, Dalian, Liaoning 116000, P. R. China;

Corresponding author：

LIU Ailian, Email: cjr.liuailian@vip.163.com

Keywords：

Deep learning; Multi-task learning; Hepatocellular carcinoma; Classification; Segmentation

DOI：

10.7507/1001-5515.202208045

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Hepatocellular carcinoma (HCC) is the most common liver malignancy, where HCC segmentation and prediction of the degree of pathological differentiation are two important tasks in surgical treatment and prognosis evaluation. Existing methods usually solve these two problems independently without considering the correlation of the two tasks. In this paper, we propose a multi-task learning model that aims to accomplish the segmentation task and classification task simultaneously. The model consists of a segmentation subnet and a classification subnet. A multi-scale feature fusion method is proposed in the classification subnet to improve the classification accuracy, and a boundary-aware attention is designed in the segmentation subnet to solve the problem of tumor over-segmentation. A dynamic weighted average multi-task loss is used to make the model achieve optimal performance in both tasks simultaneously. The experimental results of this method on 295 HCC patients are superior to other multi-task learning methods, with a Dice similarity coefficient (Dice) of (83.9 ± 0.88)% on the segmentation task, while the average recall is (86.08 ± 0.83)% and an F1 score is (80.05 ± 1.7)% on the classification task. The results show that the multi-task learning method proposed in this paper can perform the classification task and segmentation task well at the same time, which can provide theoretical reference for clinical diagnosis and treatment of HCC patients.

Citation： WEN Han, ZHAO Ying, YANG Yong, WANG Hongkai, LIU Ailian, YAO Yu, FU Zhongliang. Hepatocellular carcinoma segmentation and pathological differentiation degree prediction method based on multi-task learning. Journal of Biomedical Engineering, 2023, 40(1): 60-69. doi: 10.7507/1001-5515.202208045 Copy

1.	Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a Cancer Journal for Clinicians, 2018, 68(6): 394-424.
2.	El–Serag H B, Rudolph K L. Hepatocellular carcinoma: epidemiology and molecular carcinogenesis. Gastroenterology, 2007, 132(7): 2557-2576.
3.	Yang D W, Jia X B, Xiao Y J, et al. Noninvasive evaluation of the pathologic grade of hepatocellular carcinoma using MCF-3DCNN: a pilot study. BioMed research international, 2019, 2019: 9783106.
4.	Valanarasu J M J, Sindagi V A, Hacihaliloglu I, et al. KiU-Net: overcomplete convolutional architectures for biomedical image and volumetric segmentation. IEEE Transactions on Medical Imaging, 2022, 41(4): 965-976.
5.	Oktay O, Schlemper J, Folgoc L L, et al. Attention U-net: learning where to look for the pancreas. arXiv preprint, 2018, arXiv: 1804.03999.
6.	Hatamizadeh A, Tang Y, Nath V, et al. Unetr: transformers for 3D medical image segmentation//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 574-584.
7.	Cao H, Wang Y, Chen J, et al. Swin-unet: unet-like pure transformer for medical image segmentation. arXiv preprint, 2021, arXiv: 2105.05537.
8.	Qiu Z, Pan Y, Wei J, et al. Predicting symptoms from multiphasic MRI via multi-instance attention learning for hepatocellular carcinoma grading//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2021: 439-448.
9.	Zhou Q, Zhou Z, Chen C, et al. Grading of hepatocellular carcinoma using 3D SE-DenseNet in dynamic enhanced MR images. Computers in biology and medicine, 2019, 107: 47-57.
10.	Zhou Y, Chen H, Li Y, et al. Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound images. Medical Image Analysis, 2021, 70: 101918.
11.	Wang P, Patel V M, Hacihaliloglu I. Simultaneous segmentation and classification of bone surfaces from ultrasound using a multi-feature guided CNN//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2018: 134-142.
12.	Chen C, Bai W, Rueckert D. Multi-task learning for left atrial segmentation on GE-MRI//International Workshop on Statistical Atlases and Computational Models of the Heart. Cham: Springer, 2018: 292-301.
13.	Li C F, Xu Y D, Ding X H, et al. MultiR-Net: a novel joint learning network for COVID-19 segmentation and classification. Computers in Biology and Medicine, 2022, 144: 105340.
14.	Chakravarty A, Sivswamy J. A deep learning based joint segmentation and classification framework for glaucoma assesment in retinal color fundus images. arXiv preprint, 2018, arXiv: 1808.01355.
15.	Selvaraju R R, Cogswell M, Das A, et al. Grad-cam: visual explanations from deep networks via gradient-based localization//Proceedings of the IEEE International Conference on Computer Vision. 2017: 618-626.
16.	Liu S, Johns E, Davison A J. End-to-end multi-task learning with attention//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1871-1880.
17.	Rosiak G, Podgórska J, Rosiak E, et al. CT/MRI LI-RADS v2017–review of the guidelines. Polish Journal of Radiology, 2018, 83: e355-e365.
18.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
19.	Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 4700-4708.
20.	Carreira J, Zisserman A. Quo vadis, action recognition? A new model and the kinetics dataset//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 6299-6308.
21.	Çiçek Ö, Abdulkadir A, Lienkamp S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2016: 424-432.
22.	Milletari F, Navab N, Ahmadi S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation//2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016: 565-571.

1. Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a Cancer Journal for Clinicians, 2018, 68(6): 394-424.
2. El–Serag H B, Rudolph K L. Hepatocellular carcinoma: epidemiology and molecular carcinogenesis. Gastroenterology, 2007, 132(7): 2557-2576.
3. Yang D W, Jia X B, Xiao Y J, et al. Noninvasive evaluation of the pathologic grade of hepatocellular carcinoma using MCF-3DCNN: a pilot study. BioMed research international, 2019, 2019: 9783106.
4. Valanarasu J M J, Sindagi V A, Hacihaliloglu I, et al. KiU-Net: overcomplete convolutional architectures for biomedical image and volumetric segmentation. IEEE Transactions on Medical Imaging, 2022, 41(4): 965-976.
5. Oktay O, Schlemper J, Folgoc L L, et al. Attention U-net: learning where to look for the pancreas. arXiv preprint, 2018, arXiv: 1804.03999.
6. Hatamizadeh A, Tang Y, Nath V, et al. Unetr: transformers for 3D medical image segmentation//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 574-584.
7. Cao H, Wang Y, Chen J, et al. Swin-unet: unet-like pure transformer for medical image segmentation. arXiv preprint, 2021, arXiv: 2105.05537.
8. Qiu Z, Pan Y, Wei J, et al. Predicting symptoms from multiphasic MRI via multi-instance attention learning for hepatocellular carcinoma grading//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2021: 439-448.
9. Zhou Q, Zhou Z, Chen C, et al. Grading of hepatocellular carcinoma using 3D SE-DenseNet in dynamic enhanced MR images. Computers in biology and medicine, 2019, 107: 47-57.
10. Zhou Y, Chen H, Li Y, et al. Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound images. Medical Image Analysis, 2021, 70: 101918.
11. Wang P, Patel V M, Hacihaliloglu I. Simultaneous segmentation and classification of bone surfaces from ultrasound using a multi-feature guided CNN//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2018: 134-142.
12. Chen C, Bai W, Rueckert D. Multi-task learning for left atrial segmentation on GE-MRI//International Workshop on Statistical Atlases and Computational Models of the Heart. Cham: Springer, 2018: 292-301.
13. Li C F, Xu Y D, Ding X H, et al. MultiR-Net: a novel joint learning network for COVID-19 segmentation and classification. Computers in Biology and Medicine, 2022, 144: 105340.
14. Chakravarty A, Sivswamy J. A deep learning based joint segmentation and classification framework for glaucoma assesment in retinal color fundus images. arXiv preprint, 2018, arXiv: 1808.01355.
15. Selvaraju R R, Cogswell M, Das A, et al. Grad-cam: visual explanations from deep networks via gradient-based localization//Proceedings of the IEEE International Conference on Computer Vision. 2017: 618-626.
16. Liu S, Johns E, Davison A J. End-to-end multi-task learning with attention//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1871-1880.
17. Rosiak G, Podgórska J, Rosiak E, et al. CT/MRI LI-RADS v2017–review of the guidelines. Polish Journal of Radiology, 2018, 83: e355-e365.
18. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
19. Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 4700-4708.
20. Carreira J, Zisserman A. Quo vadis, action recognition? A new model and the kinetics dataset//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 6299-6308.
21. Çiçek Ö, Abdulkadir A, Lienkamp S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2016: 424-432.
22. Milletari F, Navab N, Ahmadi S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation//2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016: 565-571.

Journal of Biomedical Engineering

Hepatocellular carcinoma segmentation and pathological differentiation degree prediction method based on multi-task learning

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content