Research progress of artificial intelligence convolutional neural network in whole slide image analysis_Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Authors：

ZHAO Mengmeng , WANG Yang , DENG Jiajun , SHE Yunlang ,  CHEN Chang

Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University, Shanghai, 200433, P.R.China;

Corresponding author：

CHEN Chang, Email: chenthoracic@163.com

Keywords：

Whole slide image; artificial intelligence; deep learning; convolution neural network

DOI：

10.7507/1007-4848.201908034

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Histopathology is still the golden standard for the diagnosis of clinical diseases. Whole slide image (WSI) can make up for the shortcomings of traditional glass slices, such as easy damage, difficult retrieval and poor diagnostic repeatability, but it also brings huge workload. Artificial intelligence (AI) assisted pathologist's WSI analysis can solve the problem of low efficiency and improve the consistency of diagnosis. Among them, the convolution neural network (CNN) algorithm is the most widely used. This article aims to review the reported application of CNN in WSI image analysis, summarizes the development trend of CNN in the field of pathology and makes a prospect.

Citation： ZHAO Mengmeng, WANG Yang, DENG Jiajun, SHE Yunlang, CHEN Chang. Research progress of artificial intelligence convolutional neural network in whole slide image analysis. Chinese Journal of Clinical Thoracic and Cardiovascular Surgery, 2019, 26(11): 1063-1068. doi: 10.7507/1007-4848.201908034 Copy

1.	Pantanowitz L, Farahani N, Parwani A. Whole slide imaging in pathology: advantages, limitations, and emerging perspectives. Path Lab Med Inte, 2015.
2.	Higgins C. Applications and challenges of digital pathology and whole slide imaging. Biotech Histochem, 2015, 90(5): 341-347.
3.	Al Habeeb A, Evans A, Ghazarian D. Virtual microscopy using whole-slide imaging as an enabler for teledermatopathology: A paired consultant validation study. J Pathol Inform, 2012, 3: 2.
4.	Pantanowitz L, Wiley CA, Demetris A, et al. Experience with multimodality telepathology at the University of Pittsburgh Medical Center. J Pathol Inform, 2012, 3: 45.
5.	Thorstenson S, Molin J, Lundström C. Implementation of large-scale routine diagnostics using whole slide imaging in Sweden: Digital pathology experiences 2006-2013. J Pathol Inform, 2014, 5(1): 14.
6.	Al-Janabi S, Huisman A, Vink A, et al. Whole slide images for primary diagnostics in dermatopathology: a feasibility study. J Clin Pathol, 2012, 65(2): 152-158.
7.	Dawson PJ, Johnson JG, Edgemon LJ, et al. Outpatient frozen sections by telepathology in a Veterans administration medical center. Hum Pathol, 2000, 31(7): 786-788.
8.	Okada DH, Binder SW, Felten CL, et al. "Virtual microscopy" and the internet as telepathology consultation tools: diagnostic accuracy in evaluating melanocytic skin lesions. Am J Dermatopathol, 1999, 21(6): 525-531.
9.	Massone C, Soyer HP, Lozzi GP, et al. Feasibility and diagnostic agreement in teledermatopathology using a virtual slide system. Hum Pathol, 2007, 38(4): 546-554.
10.	Koch LH, Lampros JN, Delong LK, et al. Randomized comparison of virtual microscopy and traditional glass microscopy in diagnostic accuracy among dermatology and pathology residents. Hum Pathol, 2009, 40(5): 662-667.
11.	Al-Janabi S, Huisman A, Nap M, et al. Whole slide images as a platform for initial diagnostics in histopathology in a medium-sized routine laboratory. J Clin Pathol, 2012, 65(12): 1107-1111.
12.	House JC, Henderson-Jackson EB, Johnson JO, et al. Diagnostic digital cytopathology: Are we ready yet? J Pathol Inform, 2013, 4: 28.
13.	Wright AM, Smith D, Dhurandhar B, et al. Digital slide imaging in cervicovaginal cytology: a pilot study. Arch Pathol Lab Med, 2013, 137(5): 618-624.
14.	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436-444.
15.	Bi WL, Hosny A, Schabath MB, et al. Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin, 2019, 69(2): 127-157.
16.	Li X, Zhang S, Zhang Q, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol, 2019, 20(2): 193-201.
17.	Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.
18.	De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med, 2018, 24(9): 1342-1350.
19.	Coudray N, Ocampo PS, Sakellaropoulos T, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med, 2018, 24(10): 1559-1567.
20.	Wang S, Shi J, Ye Z, et al. Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. Eur Respir J, 2019, 53(3): pii: 1800986.
21.	Janowczyk A, Madabhushi A. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. J Pathol Inform, 2016, 7: 29.
22.	Campanella G, Silva VWK, Fuchs TJ. Terabyte-scale deep multiple instance learning for classification and localization in pathology. 2018, 1805. [Internet]. [cited August 10, 2019]. Available from: https://arxiv.org/abs/1805.06983.
23.	Kwak JT, Hewitt SM. Multiview boosting digital pathology analysis of prostate cancer. Comput Methods Programs Biomed, 2017, 142: 91-99.
24.	Rannen Triki A, Blaschko MB, Jung YM, et al. Intraoperative margin assessment of human breast tissue in optical coherence tomography images using deep neural networks. Comput Med Imaging Graph, 2018, 69: 21-32.
25.	Garud H, Karri SPK, Sheet D, et al. High-magnification multi-views based classification of breast fine needle aspiration cytology cell samples using fusion of decisions from deep convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017: 828-833.
26.	Ha R, Mutasa S, Sant EPV, et al. Accuracy of distinguishing atypical ductal hyperplasia from ductal carcinoma in situ with convolutional neural network-based machine learning approach using mammographic image data. AJR Am J Roentgenol, 2019, [Epub ahead of print].
27.	Mercan E, Mehta S, Bartlett J, et al. Assessment of machine learning of breast pathology structures for automated differentiation of breast cancer and high-risk proliferative lesions. JAMA Netw Open, 2019, 2(8): e198777.
28.	Gertych A, Swiderska-Chadaj Z, Ma Z, et al. Convolutional neural networks can accurately distinguish four histologic growth patterns of lung adenocarcinoma in digital slides. Sci Rep, 2019, 9(1): 1483.
29.	Wei JW, Tafe LJ, Linnik YA, et al. Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks. Sci Rep, 2019, 9(1): 3358.
30.	Lodha S, Saggar S, Celebi JT, et al. Discordance in the histopathologic diagnosis of difficult melanocytic neoplasms in the clinical setting. J Cutan Pathol, 2008, 35(4): 349-352.
31.	Hekler A, Utikal JS, Enk AH, et al. Pathologist-level classification of histopathological melanoma images with deep neural networks. Eur J Cancer, 2019, 115: 79-83.
32.	Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med, 2019, 25(8): 1301-1309.
33.	Wu M, Yan C, Liu H, et al. Automatic classification of ovarian cancer types from cytological images using deep convolutional neural networks. Biosci Rep, 2018, 38(3): pii: BSR20180289.
34.	Ertosun MG, Rubin DL. Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. AMIA Annu Symp Proc, 2015, 2015: 1899-1908.
35.	Zhang L, Le Lu, Nogues I, et al. DeepPap: deep convolutional networks for cervical cell classification. IEEE J Biomed Health Inform, 2017, 21(6): 1633-1643.
36.	Sertel O, Kong J, Shimada H, et al. Computer-aided prognosis of neuroblastoma on whole-slide images: classification of stromal development. Pattern Recognit, 2009, 42(6): 1093-1103.
37.	Sabo E, Beck AH, Montgomery EA, et al. Computerized morphometry as an aid in determining the grade of dysplasia and progression to adenocarcinoma in Barrett's esophagus. Lab Invest, 2006, 86(12): 1261-1271.
38.	Litjens G, Sánchez CI, Timofeeva N, et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci Rep, 2016, 6: 26286.
39.	Ehteshami Bejnordi B, Veta M, Johannes van Diest P, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 2017, 318(22): 2199-2210.
40.	Li Y, Ping WJa CV, Recognition P. Cancer metastasis detection with neural conditional random field. 2018, 1806. [Internet]. [cited August 10, 2019]. Available from: https://arxiv.org/abs/1806.07064.
41.	Steiner DF, MacDonald R, Liu Y, et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol, 2018, 42(12): 1636-1646.
42.	Kather JN, Krisam J, Charoentong P, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med, 2019, 16(1): e1002730.
43.	Veta M, Heng YJ, Stathonikos N, et al. Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge. Med Image Anal, 2019, 54: 111-121.
44.	Harder N, Schönmeyer R, Nekolla K, et al. Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma. Sci Rep, 2019, 9(1): 7449.
45.	Zhou N, Fedorov A, Fennessy FM, et al. Large scale digital prostate pathology image analysis combining feature extraction and deep neural network. 2017. [Internet]. arXiv: Computer Vision and Pattern Recognition, 2017, 1705. [cited August 10, 2019]. Available from: https://arxiv.org/abs/1705.02678v2.
46.	Yu KH, Zhang C, Berry GJ, et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat Commun, 2016, 7: 12474.
47.	Beck AH, Sangoi AR, Leung S, et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med, 2011, 3(108): 108ra113.
48.	Sertel O, Kong J, Catalyurek UV, et al. Histopathological image analysis using model-based intermediate representations and color texture: follicular lymphoma grading. J Sign Pro Sys, 2008, 55(1-3): 169-183.

1. Pantanowitz L, Farahani N, Parwani A. Whole slide imaging in pathology: advantages, limitations, and emerging perspectives. Path Lab Med Inte, 2015.
2. Higgins C. Applications and challenges of digital pathology and whole slide imaging. Biotech Histochem, 2015, 90(5): 341-347.
3. Al Habeeb A, Evans A, Ghazarian D. Virtual microscopy using whole-slide imaging as an enabler for teledermatopathology: A paired consultant validation study. J Pathol Inform, 2012, 3: 2.
4. Pantanowitz L, Wiley CA, Demetris A, et al. Experience with multimodality telepathology at the University of Pittsburgh Medical Center. J Pathol Inform, 2012, 3: 45.
5. Thorstenson S, Molin J, Lundström C. Implementation of large-scale routine diagnostics using whole slide imaging in Sweden: Digital pathology experiences 2006-2013. J Pathol Inform, 2014, 5(1): 14.
6. Al-Janabi S, Huisman A, Vink A, et al. Whole slide images for primary diagnostics in dermatopathology: a feasibility study. J Clin Pathol, 2012, 65(2): 152-158.
7. Dawson PJ, Johnson JG, Edgemon LJ, et al. Outpatient frozen sections by telepathology in a Veterans administration medical center. Hum Pathol, 2000, 31(7): 786-788.
8. Okada DH, Binder SW, Felten CL, et al. "Virtual microscopy" and the internet as telepathology consultation tools: diagnostic accuracy in evaluating melanocytic skin lesions. Am J Dermatopathol, 1999, 21(6): 525-531.
9. Massone C, Soyer HP, Lozzi GP, et al. Feasibility and diagnostic agreement in teledermatopathology using a virtual slide system. Hum Pathol, 2007, 38(4): 546-554.
10. Koch LH, Lampros JN, Delong LK, et al. Randomized comparison of virtual microscopy and traditional glass microscopy in diagnostic accuracy among dermatology and pathology residents. Hum Pathol, 2009, 40(5): 662-667.
11. Al-Janabi S, Huisman A, Nap M, et al. Whole slide images as a platform for initial diagnostics in histopathology in a medium-sized routine laboratory. J Clin Pathol, 2012, 65(12): 1107-1111.
12. House JC, Henderson-Jackson EB, Johnson JO, et al. Diagnostic digital cytopathology: Are we ready yet? J Pathol Inform, 2013, 4: 28.
13. Wright AM, Smith D, Dhurandhar B, et al. Digital slide imaging in cervicovaginal cytology: a pilot study. Arch Pathol Lab Med, 2013, 137(5): 618-624.
14. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436-444.
15. Bi WL, Hosny A, Schabath MB, et al. Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin, 2019, 69(2): 127-157.
16. Li X, Zhang S, Zhang Q, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol, 2019, 20(2): 193-201.
17. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.
18. De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med, 2018, 24(9): 1342-1350.
19. Coudray N, Ocampo PS, Sakellaropoulos T, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med, 2018, 24(10): 1559-1567.
20. Wang S, Shi J, Ye Z, et al. Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. Eur Respir J, 2019, 53(3): pii: 1800986.
21. Janowczyk A, Madabhushi A. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. J Pathol Inform, 2016, 7: 29.
22. Campanella G, Silva VWK, Fuchs TJ. Terabyte-scale deep multiple instance learning for classification and localization in pathology. 2018, 1805. [Internet]. [cited August 10, 2019]. Available from: https://arxiv.org/abs/1805.06983.
23. Kwak JT, Hewitt SM. Multiview boosting digital pathology analysis of prostate cancer. Comput Methods Programs Biomed, 2017, 142: 91-99.
24. Rannen Triki A, Blaschko MB, Jung YM, et al. Intraoperative margin assessment of human breast tissue in optical coherence tomography images using deep neural networks. Comput Med Imaging Graph, 2018, 69: 21-32.
25. Garud H, Karri SPK, Sheet D, et al. High-magnification multi-views based classification of breast fine needle aspiration cytology cell samples using fusion of decisions from deep convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017: 828-833.
26. Ha R, Mutasa S, Sant EPV, et al. Accuracy of distinguishing atypical ductal hyperplasia from ductal carcinoma in situ with convolutional neural network-based machine learning approach using mammographic image data. AJR Am J Roentgenol, 2019, [Epub ahead of print].
27. Mercan E, Mehta S, Bartlett J, et al. Assessment of machine learning of breast pathology structures for automated differentiation of breast cancer and high-risk proliferative lesions. JAMA Netw Open, 2019, 2(8): e198777.
28. Gertych A, Swiderska-Chadaj Z, Ma Z, et al. Convolutional neural networks can accurately distinguish four histologic growth patterns of lung adenocarcinoma in digital slides. Sci Rep, 2019, 9(1): 1483.
29. Wei JW, Tafe LJ, Linnik YA, et al. Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks. Sci Rep, 2019, 9(1): 3358.
30. Lodha S, Saggar S, Celebi JT, et al. Discordance in the histopathologic diagnosis of difficult melanocytic neoplasms in the clinical setting. J Cutan Pathol, 2008, 35(4): 349-352.
31. Hekler A, Utikal JS, Enk AH, et al. Pathologist-level classification of histopathological melanoma images with deep neural networks. Eur J Cancer, 2019, 115: 79-83.
32. Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med, 2019, 25(8): 1301-1309.
33. Wu M, Yan C, Liu H, et al. Automatic classification of ovarian cancer types from cytological images using deep convolutional neural networks. Biosci Rep, 2018, 38(3): pii: BSR20180289.
34. Ertosun MG, Rubin DL. Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. AMIA Annu Symp Proc, 2015, 2015: 1899-1908.
35. Zhang L, Le Lu, Nogues I, et al. DeepPap: deep convolutional networks for cervical cell classification. IEEE J Biomed Health Inform, 2017, 21(6): 1633-1643.
36. Sertel O, Kong J, Shimada H, et al. Computer-aided prognosis of neuroblastoma on whole-slide images: classification of stromal development. Pattern Recognit, 2009, 42(6): 1093-1103.
37. Sabo E, Beck AH, Montgomery EA, et al. Computerized morphometry as an aid in determining the grade of dysplasia and progression to adenocarcinoma in Barrett's esophagus. Lab Invest, 2006, 86(12): 1261-1271.
38. Litjens G, Sánchez CI, Timofeeva N, et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci Rep, 2016, 6: 26286.
39. Ehteshami Bejnordi B, Veta M, Johannes van Diest P, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 2017, 318(22): 2199-2210.
40. Li Y, Ping WJa CV, Recognition P. Cancer metastasis detection with neural conditional random field. 2018, 1806. [Internet]. [cited August 10, 2019]. Available from: https://arxiv.org/abs/1806.07064.
41. Steiner DF, MacDonald R, Liu Y, et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol, 2018, 42(12): 1636-1646.
42. Kather JN, Krisam J, Charoentong P, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med, 2019, 16(1): e1002730.
43. Veta M, Heng YJ, Stathonikos N, et al. Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge. Med Image Anal, 2019, 54: 111-121.
44. Harder N, Schönmeyer R, Nekolla K, et al. Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma. Sci Rep, 2019, 9(1): 7449.
45. Zhou N, Fedorov A, Fennessy FM, et al. Large scale digital prostate pathology image analysis combining feature extraction and deep neural network. 2017. [Internet]. arXiv: Computer Vision and Pattern Recognition, 2017, 1705. [cited August 10, 2019]. Available from: https://arxiv.org/abs/1705.02678v2.
46. Yu KH, Zhang C, Berry GJ, et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat Commun, 2016, 7: 12474.
47. Beck AH, Sangoi AR, Leung S, et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med, 2011, 3(108): 108ra113.
48. Sertel O, Kong J, Catalyurek UV, et al. Histopathological image analysis using model-based intermediate representations and color texture: follicular lymphoma grading. J Sign Pro Sys, 2008, 55(1-3): 169-183.

Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Research progress of artificial intelligence convolutional neural network in whole slide image analysis

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content