Validation of multivariate selection method in clinical prediction models: based on MIMIC database_Chinese Journal of Evidence-Based Medicine

Authors：

ZHENG Shuai ^1,2 , HUANG Tao ¹ , YANG Rui ¹ , LI Li ¹ , QIAO Mengmeng ² , CHEN Chong ² ,  LYU Jun ¹

1. Department of Clinical Research, the First Affiliated Hospital of Jinan University, Guangzhou 510630, P.R.China;
2. School of Public Health, Shanxi University of Chinese Medicine, Xianyang 712046, P.R.China;

Corresponding author：

LYU Jun, Email: lyujun2020@jnu.edu.cn

Keywords：

Variable selection; Clinical prediction model; MIMIC database; Model establishment

DOI：

10.7507/1672-2531.202107175

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To verify the influence of different variable selection methods on the performance of clinical prediction models. Methods Three sample sets were extracted from the MIMIC database (acute myocardial infarction group, sepsis group, and cerebral hemorrhage group) using the direct entry of COX regression, step by step forward, step by step backward, LASSO, and ridge regression, based on random forest. These existing six methods of variable importance algorithm, and the optimal variable set of different selected methods were used to construct the model. Through the C index, the area under the ROC curve (AUC value) and the calibration curve, and the results within and between groups were compared. Results The variables and numbers selected by the six variable selection methods were different, however, whether it was within or between groups did not reflect which method had the advantage of significantly improving the performance of the model. Conclusions Prior to using the variable selection method to establish a clinical prediction model, we should first clarify the research purpose and determine the type of data. Combining medical knowledge to select a method that can meet the data type and simultaneously achieve the research purpose.

Citation： ZHENG Shuai, HUANG Tao, YANG Rui, LI Li, QIAO Mengmeng, CHEN Chong, LYU Jun. Validation of multivariate selection method in clinical prediction models: based on MIMIC database. Chinese Journal of Evidence-Based Medicine, 2021, 21(12): 1463-1467. doi: 10.7507/1672-2531.202107175 Copy

1.	Tripepi G, Jager KJ, Dekker FW, et al. Testing for causality and prognosis: etiological and prognostic models. Kidney Int, 2008, 74(12): 1512-1515.
2.	von Düring ME, Jenssen T, Bollerslev J, et al. Visceral fat is better related to impaired glucose metabolism than body mass index after kidney transplantation. Transpl Int, 2015, 28(10): 1162-1171.
3.	Bhat M, Hathcock M, Kremers WK, et al. Portal vein encasement predicts neoadjuvant therapy response in liver transplantation for perihilar cholangiocarcinoma protocol. Transpl Int, 2015, 28(12): 1383-1391.
4.	Pianta TJ, Peake PW, Pickering JW, et al. Evaluation of biomarkers of cell cycle arrest and inflammation in prediction of dialysis or recovery after kidney transplantation. Transpl Int, 2015, 28(12): 1392-1404.
5.	Núñez E, Steyerberg EW, Núñez J. Regression modeling strategies. Rev Esp Cardiol, 2011, 64(6): 501-507.
6.	Chowdhury MZI, Turin TC. Variable selection strategies and its importance in clinical prediction modelling. Fam Med Community Health, 2020, 8(1): e000262.
7.	Johnson AE, Pollard TJ, Shen L, et al. MIMIC-Ⅲ, a freely accessible critical care database. Sci Data, 2016, 3: 160035.
8.	Yang J, Li Y, Liu Q, et al. Brief introduction of medical database and data mining technology in big data era. J Evid Based Med, 2020, 13(1): 57-69.
9.	Guyon I, Elisseeff A. An introduction to variable and feature selection. J Mach Learn Res, 2003, 3(3): 1157-1182.
10.	Royston P, Moons KG, Altman DG, et al. Prognosis and prognostic research: Developing a prognostic model. BMJ, 2009, 338: b604.
11.	Heinze G, Dunkler D. Five myths about variable selection. Transpl Int, 2017, 30(1): 6-10.
12.	Zhang Z, Reinikainen J, Adeleke KA, et al. Time-varying covariates and coefficients in Cox regression models. Ann Transl Med, 2018, 6(7): 121.
13.	Alves LF, Fernandes BF, Burnier JV, et al. Incidence of epithelial lesions of the conjunctiva in a review of 12 102 specimens in Canada (Quebec). Arq Bras Oftalmol, 2011, 74(1): 21-23.
14.	张玉. 自变量个数远大于样本数情形下(p>n)罚函数回归法的改进. 江苏教育学院学报(自然科学版), 2012, 28(3): 28-32.
15.	Kursa M, Rudnicki W. Feature selection with the boruta package. J Stat Soft, 2010, 36: 1-13.
16.	Goodman S. A dirty dozen: twelve p-value misconceptions. Semin Hematol, 2008, 45(3): 135-140.
17.	Han J, Zheng H, Xing Y, et al. V2V: A deep learning approach to variable-to-variable selection and translation for multivariate time-varying data. IEEE Trans Vis Comput Graph, 2021, 27(2): 1290-1300.
18.	Degenhardt F, Seifert S, Szymczak S. Evaluation of variable selection methods for random forests and omics data sets. Brief Bioinform, 2019, 20(2): 492-503.
19.	Heinze G, Wallisch C, Dunkler D. Variable selection - a review and recommendations for the practicing statistician. Biom J, 2018, 60(3): 431-449.
20.	Wiegand RE. Performance of using multiple stepwise algorithms for variable selection. Stat Med, 2010, 29(15): 1647-1659.

1. Tripepi G, Jager KJ, Dekker FW, et al. Testing for causality and prognosis: etiological and prognostic models. Kidney Int, 2008, 74(12): 1512-1515.
2. von Düring ME, Jenssen T, Bollerslev J, et al. Visceral fat is better related to impaired glucose metabolism than body mass index after kidney transplantation. Transpl Int, 2015, 28(10): 1162-1171.
3. Bhat M, Hathcock M, Kremers WK, et al. Portal vein encasement predicts neoadjuvant therapy response in liver transplantation for perihilar cholangiocarcinoma protocol. Transpl Int, 2015, 28(12): 1383-1391.
4. Pianta TJ, Peake PW, Pickering JW, et al. Evaluation of biomarkers of cell cycle arrest and inflammation in prediction of dialysis or recovery after kidney transplantation. Transpl Int, 2015, 28(12): 1392-1404.
5. Núñez E, Steyerberg EW, Núñez J. Regression modeling strategies. Rev Esp Cardiol, 2011, 64(6): 501-507.
6. Chowdhury MZI, Turin TC. Variable selection strategies and its importance in clinical prediction modelling. Fam Med Community Health, 2020, 8(1): e000262.
7. Johnson AE, Pollard TJ, Shen L, et al. MIMIC-Ⅲ, a freely accessible critical care database. Sci Data, 2016, 3: 160035.
8. Yang J, Li Y, Liu Q, et al. Brief introduction of medical database and data mining technology in big data era. J Evid Based Med, 2020, 13(1): 57-69.
9. Guyon I, Elisseeff A. An introduction to variable and feature selection. J Mach Learn Res, 2003, 3(3): 1157-1182.
10. Royston P, Moons KG, Altman DG, et al. Prognosis and prognostic research: Developing a prognostic model. BMJ, 2009, 338: b604.
11. Heinze G, Dunkler D. Five myths about variable selection. Transpl Int, 2017, 30(1): 6-10.
12. Zhang Z, Reinikainen J, Adeleke KA, et al. Time-varying covariates and coefficients in Cox regression models. Ann Transl Med, 2018, 6(7): 121.
13. Alves LF, Fernandes BF, Burnier JV, et al. Incidence of epithelial lesions of the conjunctiva in a review of 12 102 specimens in Canada (Quebec). Arq Bras Oftalmol, 2011, 74(1): 21-23.
14. 张玉. 自变量个数远大于样本数情形下(p>n)罚函数回归法的改进. 江苏教育学院学报(自然科学版), 2012, 28(3): 28-32.
15. Kursa M, Rudnicki W. Feature selection with the boruta package. J Stat Soft, 2010, 36: 1-13.
16. Goodman S. A dirty dozen: twelve p-value misconceptions. Semin Hematol, 2008, 45(3): 135-140.
17. Han J, Zheng H, Xing Y, et al. V2V: A deep learning approach to variable-to-variable selection and translation for multivariate time-varying data. IEEE Trans Vis Comput Graph, 2021, 27(2): 1290-1300.
18. Degenhardt F, Seifert S, Szymczak S. Evaluation of variable selection methods for random forests and omics data sets. Brief Bioinform, 2019, 20(2): 492-503.
19. Heinze G, Wallisch C, Dunkler D. Variable selection - a review and recommendations for the practicing statistician. Biom J, 2018, 60(3): 431-449.
20. Wiegand RE. Performance of using multiple stepwise algorithms for variable selection. Stat Med, 2010, 29(15): 1647-1659.

Chinese Journal of Evidence-Based Medicine

Validation of multivariate selection method in clinical prediction models: based on MIMIC database

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content