Review of research on detection and tracking of minimally invasive surgical tools based on deep learning_Journal of Biomedical Engineering

Authors：

LIU Yuying ,  ZHAO Zijian

School of Control Science and Engineering, Shandong University, Jinan 250061, P.R.China;

Corresponding author：

Keywords：

deep learning; convolutional neural network; fully supervised; weakly supervised; surgical tool detection and tracking

DOI：

10.7507/1001-5515.201904061

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The application of minimally invasive surgical tool detection and tracking technology based on deep learning in minimally invasive surgery is currently a research hotspot. This paper firstly expounds the relevant technical content of the minimally invasive surgery tool detection and tracking, which mainly introduces the advantages based on deep learning algorithm. Then, this paper summarizes the algorithm for detection and tracking surgical tools based on fully supervised deep neural network and the emerging algorithm for detection and tracking surgical tools based on weakly supervised deep neural network. Several typical algorithm frameworks and their flow charts based on deep convolutional and recurrent neural networks are summarized emphatically, so as to enable researchers in relevant fields to understand the current research progress more systematically and provide reference for minimally invasive surgeons to select navigation technology. In the end, this paper provides a general direction for the further research of minimally invasive surgical tool detection and tracking technology based on deep learning.

Citation： LIU Yuying, ZHAO Zijian. Review of research on detection and tracking of minimally invasive surgical tools based on deep learning. Journal of Biomedical Engineering, 2019, 36(5): 870-878. doi: 10.7507/1001-5515.201904061 Copy

1.	Zhao Zijian, Voros S, Weng Ying, et al. Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method. Computer Assisted Surgery, 2017, 22(1): 26-35.
2.	Bouget D, Allan M, Stoyanov D, et al. Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal, 2017, 35: 633-654.
3.	Du Xiaofei, Allan M, Dore A, et al. Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery. Int J Comput Assist Radiol Surg, 2016, 11(6): 1109-1119.
4.	Twinanda A P, Shehata S M, Mutter D, et al. EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging, 2017, 36(1): 86-97.
5.	Rieke N, Tan D J, Tombari F, et al. Real-time online adaption for robust instrument tracking and pose estimation. Medical Image Computing and Computer Assisted Intervention (MICCAI 2016), Springer, 2016: 422-430.
6.	Sahu M, Moerman D, Mewes P, et al. Instrument state recognition and tracking for effective control of robotized laparoscopic systems. International Journal of Mechanical Engineering and Robotics Research, 2016, 5(1): 33-38.
7.	García-Peraza-Herrera L C, Li Wenqi, Fidon L, et al. ToolNet: holistically-nested real-time segmentation of robotic surgical tools//IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2017: 5717-5722.
8.	Chen Zhaorui, Zhao Zijian, Cheng Xiaolin. Surgical instruments tracking based on deep learning with lines detection and spatio-temporal context//Chinese Automation Congress (CAC 2017). IEEE, 2018. DOI: 10.1109/CAC.2017.8243236.
9.	Sahu M, Mukhopadhyay A, Szengel A, et al. Addressing multi-label imbalance problem of surgical tool detection using CNN. Int J Comput Assist Radiol Surg, 2017, 12(6): 1013-1020.
10.	García-Peraza-Herrera L C, Li Wenqi, Gruijthuijsen C, et al. Real-time segmentation of non-rigid surgical tools based on deep learning and tracking// Computer-Assisted and Robotic Endoscopy (CARE 2016). Springer, 2016: 84-95.
11.	Zhao Z, Voros S, Chen Z, et al. Surgical tool tracking based on two CNNs: from coarse to fine. The Journal of Engineering, 2019(14): 467-472.
12.	Zhao Zijian, Chen Zhaorui, Voros S, et al. Real-time tracking of surgical instruments based on spatio-temporal context and deep learning. Computer Assisted Surgery, 2019: 1-10.
13.	Sarikaya D, Corso J J, Guru K A. Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging, 2017, 36(7): 1542-1549.
14.	Jin A, Yeung S, Jopling J, et al. Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks//31st Conference on Neural Information Processing Systems (NIPS 2017), 2018. arXiv: 1802.08774v1.
15.	Choi B, Jo K, Choi S, et al. Surgical-tools detection based on convolutional neural network in laparoscopic robot-assisted surgery//39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2017. DOI: 10.1109/embc.2017.8037183.
16.	Redmon J, Divvala S K, Girshick R B, et al. You only look once: unified, real-time object detection// Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016. arXiv: 1506.02640v5.
17.	Kurmann T, Neila P M, Du Xiaofei, et al. Simultaneous recognition and pose estimation of instruments in minimally invasive surgery//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017), Springer, 2017: 505-513.
18.	Gao Cong, Unberath M, Taylor R H, et al. Localizing dexterous surgical tools in X-ray for image-based navigation. arXiv: Computer Vision and Pattern Recognition, 2019. arXiv: 1901.06672v2.
19.	Du X, Kurmann T, Chang P L, et al. Articulated Multi-Instrument 2D pose estimation using fully convolutional networks. IEEE Trans Med Imaging, 2018, 37(5): 1276-1287.
20.	Laina I, Rieke N, Rupprecht C, et al. Concurrent segmentation and localization for tracking of surgical instruments//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017), Springer, 2017: 664-672.
21.	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016. arXiv: 1512.03385v1.
22.	Mishra K, Sathish R, Sheet D. Learning latent temporal connectionism of deep residual visual abstractions for identifying surgical tools in laparoscopy procedures//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017: 2233-2240.
23.	Hajj H A, Lamard M, Conze P, et al. Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. Med Image Anal, 2018, 47: 203-218.
24.	Hwang S, Kim H E. Self-transfer learning for weakly supervised lesion localization// Medical Image Computing and Computer-Assisted Intervention (MICCAI 2016). 2016: 239-246.
25.	Jia Zhipeng, Huang Xingyi, Chang E I, et al. Constrained deep weak supervision for histopathology image segmentation. IEEE Trans Med Imaging, 2017, 36(11): 2376-2388.
26.	Singh K K, Lee Y J. Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization// IEEE International Conference on Computer Vision (ICCV), 2017: 3524-3533.
27.	Kim D, Cho D, Yoo D, et al. Two-phase learning for weakly supervised object localization//IEEE Computer Society and the Computer Vision Foundation (CVF), 2017: 3554-3563. arXiv: 1708.02108v3.
28.	Vardazaryan A, Mutter D, Marescaux J, et al. Weakly-supervised learning for tool localization in laparoscopic videos//LABELS 2018, CVII 2018, STENT 2018: Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, 2018: 169-179. arXiv: 1806.05573v2.
29.	Nwoye C I, Mutter D, Marescaux J A. Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. Int J Comput Assist Radiol Surg, 2019, 14(6): 1059-1067.

1. Zhao Zijian, Voros S, Weng Ying, et al. Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method. Computer Assisted Surgery, 2017, 22(1): 26-35.
2. Bouget D, Allan M, Stoyanov D, et al. Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal, 2017, 35: 633-654.
3. Du Xiaofei, Allan M, Dore A, et al. Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery. Int J Comput Assist Radiol Surg, 2016, 11(6): 1109-1119.
4. Twinanda A P, Shehata S M, Mutter D, et al. EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging, 2017, 36(1): 86-97.
5. Rieke N, Tan D J, Tombari F, et al. Real-time online adaption for robust instrument tracking and pose estimation. Medical Image Computing and Computer Assisted Intervention (MICCAI 2016), Springer, 2016: 422-430.
6. Sahu M, Moerman D, Mewes P, et al. Instrument state recognition and tracking for effective control of robotized laparoscopic systems. International Journal of Mechanical Engineering and Robotics Research, 2016, 5(1): 33-38.
7. García-Peraza-Herrera L C, Li Wenqi, Fidon L, et al. ToolNet: holistically-nested real-time segmentation of robotic surgical tools//IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2017: 5717-5722.
8. Chen Zhaorui, Zhao Zijian, Cheng Xiaolin. Surgical instruments tracking based on deep learning with lines detection and spatio-temporal context//Chinese Automation Congress (CAC 2017). IEEE, 2018. DOI: 10.1109/CAC.2017.8243236.
9. Sahu M, Mukhopadhyay A, Szengel A, et al. Addressing multi-label imbalance problem of surgical tool detection using CNN. Int J Comput Assist Radiol Surg, 2017, 12(6): 1013-1020.
10. García-Peraza-Herrera L C, Li Wenqi, Gruijthuijsen C, et al. Real-time segmentation of non-rigid surgical tools based on deep learning and tracking// Computer-Assisted and Robotic Endoscopy (CARE 2016). Springer, 2016: 84-95.
11. Zhao Z, Voros S, Chen Z, et al. Surgical tool tracking based on two CNNs: from coarse to fine. The Journal of Engineering, 2019(14): 467-472.
12. Zhao Zijian, Chen Zhaorui, Voros S, et al. Real-time tracking of surgical instruments based on spatio-temporal context and deep learning. Computer Assisted Surgery, 2019: 1-10.
13. Sarikaya D, Corso J J, Guru K A. Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging, 2017, 36(7): 1542-1549.
14. Jin A, Yeung S, Jopling J, et al. Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks//31st Conference on Neural Information Processing Systems (NIPS 2017), 2018. arXiv: 1802.08774v1.
15. Choi B, Jo K, Choi S, et al. Surgical-tools detection based on convolutional neural network in laparoscopic robot-assisted surgery//39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2017. DOI: 10.1109/embc.2017.8037183.
16. Redmon J, Divvala S K, Girshick R B, et al. You only look once: unified, real-time object detection// Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016. arXiv: 1506.02640v5.
17. Kurmann T, Neila P M, Du Xiaofei, et al. Simultaneous recognition and pose estimation of instruments in minimally invasive surgery//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017), Springer, 2017: 505-513.
18. Gao Cong, Unberath M, Taylor R H, et al. Localizing dexterous surgical tools in X-ray for image-based navigation. arXiv: Computer Vision and Pattern Recognition, 2019. arXiv: 1901.06672v2.
19. Du X, Kurmann T, Chang P L, et al. Articulated Multi-Instrument 2D pose estimation using fully convolutional networks. IEEE Trans Med Imaging, 2018, 37(5): 1276-1287.
20. Laina I, Rieke N, Rupprecht C, et al. Concurrent segmentation and localization for tracking of surgical instruments//Medical Image Computing and Computer-Assisted Intervention (MICCAI 2017), Springer, 2017: 664-672.
21. He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016. arXiv: 1512.03385v1.
22. Mishra K, Sathish R, Sheet D. Learning latent temporal connectionism of deep residual visual abstractions for identifying surgical tools in laparoscopy procedures//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017: 2233-2240.
23. Hajj H A, Lamard M, Conze P, et al. Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. Med Image Anal, 2018, 47: 203-218.
24. Hwang S, Kim H E. Self-transfer learning for weakly supervised lesion localization// Medical Image Computing and Computer-Assisted Intervention (MICCAI 2016). 2016: 239-246.
25. Jia Zhipeng, Huang Xingyi, Chang E I, et al. Constrained deep weak supervision for histopathology image segmentation. IEEE Trans Med Imaging, 2017, 36(11): 2376-2388.
26. Singh K K, Lee Y J. Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization// IEEE International Conference on Computer Vision (ICCV), 2017: 3524-3533.
27. Kim D, Cho D, Yoo D, et al. Two-phase learning for weakly supervised object localization//IEEE Computer Society and the Computer Vision Foundation (CVF), 2017: 3554-3563. arXiv: 1708.02108v3.
28. Vardazaryan A, Mutter D, Marescaux J, et al. Weakly-supervised learning for tool localization in laparoscopic videos//LABELS 2018, CVII 2018, STENT 2018: Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, 2018: 169-179. arXiv: 1806.05573v2.
29. Nwoye C I, Mutter D, Marescaux J A. Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. Int J Comput Assist Radiol Surg, 2019, 14(6): 1059-1067.

Previous Article
Application of nanodrug carriers in the prevention and treatment of infection around orthopedic prosthesis
Next Article
A review of machine learning in tumor radiotherapy

Journal of Biomedical Engineering

Review of research on detection and tracking of minimally invasive surgical tools based on deep learning

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content