Methods and procedures of clinical predictive model_Chinese Journal of Evidence-Based Medicine

Authors：

MO Hangfeng ¹ , CHEN Yaping ² , HAN Hui ³ , ZHANG Yaping ⁴ , LIU Yujin ^1,5 , ZHANG Mei ¹ , LU Xiaodan ¹ , HUA Yuting ¹ , ZHU Yujia ³ , CAI Tingting ¹ , XIA Yunhui ¹ ,  SHEN Jiantong ^1,6

1. School of Medicine, Huzhou University, Huzhou 313000, P. R. China;
2. Huzhou Wuxing District Daxi Hospital, Huzhou 313000, P. R. China;
3. The First Affiliated Hospital of Huzhou University, Huzhou 313000, P. R. China;
4. Huzhou Central Hospital, Huzhou 313000, P. R. China;
5. The Second Hospital of Jinhua, Jinhua 321016, P. R. China;
6. Huzhou Precision Prevention and Control Key Laboratory of Significant Chronic Diseases, Huzhou 313000, P. R. China;

Corresponding author：

SHEN Jiantong, Email: sjiantong@163.com

Keywords：

Clinical prediction model; Diagnostic model; Prognostic model; Study design

DOI：

10.7507/1672-2531.202308135

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The use of clinical predictive modeling to guide clinical decision-making and thus provide accurate diagnosis and treatment services for patients has become a clinical consensus and trend. However, the models available for clinical use are more limited due to unstandardised research methods and poor quality of evidence. This paper introduces the development process of clinical prediction models from six aspects, data collection, model development, performance evaluation, model validation, model presentation and model updating, as well as the clinical prediction model research report statement and risk of bias assessment tools in order to provide methodological references for domestic researchers.

Citation： MO Hangfeng, CHEN Yaping, HAN Hui, ZHANG Yaping, LIU Yujin, ZHANG Mei, LU Xiaodan, HUA Yuting, ZHU Yujia, CAI Tingting, XIA Yunhui, SHEN Jiantong. Methods and procedures of clinical predictive model. Chinese Journal of Evidence-Based Medicine, 2024, 24(2): 228-236. doi: 10.7507/1672-2531.202308135 Copy

1.	Steyerberg EW, Moons KG, van der Windt DA, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med, 2013, 10(2): e1001381.
2.	Moons KG, Royston P, Vergouwe Y, et al. Prognosis and prognostic research: what, why, and how. BMJ, 2009, 338: b375.
3.	谷鸿秋, 周支瑞, 章仲恒, 等. 临床预测模型: 基本概念、应用场景及研究思路. 中国循证心血管医学杂志, 2018, 10(12): 1454-1456, 1462.
4.	Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. Cham: Springer International Publishing, 2019.
5.	Santos R, Coleman HG, Cairnduff V, et al. Clinical prediction models for pancreatic cancer in general and at-risk populations: a systematic review. Am J Gastroenterol, 2023, 118(1): 26-40.
6.	Buttia C, Llanaj E, Raeisi-Dehkordi H, et al. Prognostic models in COVID-19 infection that predict severity: a systematic review. Eur J Epidemiol, 2023, 38(4): 355-372.
7.	Kostopoulos G, Doundoulakis I, Toulis KA, et al. Prognostic models for heart failure in patients with type 2 diabetes: a systematic review and meta-analysis. Heart, 2023, 109(19): 1436-1442.
8.	Pladet LCA, Barten JMM, Vernooij LM, et al. Prognostic models for mortality risk in patients requiring ECMO. Intensive Care Med, 2023, 49(2): 131-141.
9.	Ohyama Y, Iwamura T, Hoshino T, et al. Prognostic models of quality of life after total knee replacement: a systematic review. Physiother Theory Pract, 2023.
10.	Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. BMJ, 2020: m1328.
11.	Moons KG, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med, 2015, 162(1): W1-W73.
12.	Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med, 2006, 144(3): 201-209.
13.	Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Diabet Med, 2015, 32(2): 146-154.
14.	Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med, 2014, 11(10): e1001744.
15.	Pajouheshnia R, Groenwold RHH, Peelen LM, et al. When and how to use data from randomised trials to develop or validate prognostic models. BMJ, 2019, 365: l2154.
16.	Harrell FE. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Cham: Springer International Publishing, 2015.
17.	Hemingway H. Prognosis research: why is Dr. Lydgate still waiting? J Clin Epidemiol, 2006, 59(12): 1229-1238.
18.	Clark TG, Bradburn MJ, Love SB, et al. Survival analysis part I: basic concepts and first analyses. Br J Cancer, 2003, 89(2): 232-238.
19.	Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med, 2019, 38(7): 1276-1296.
20.	van Smeden M, Moons KG, de Groot JA, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res, 2019, 28(8): 2455-2474.
21.	Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ, 2020, 368: m441.
22.	Little RJA, Rubin DB. Statistical analysis with missing data (3rd edition). Hoboken: Wiley, 2020.
23.	Graham JW. Missing data analysis: making it work in the real world. Annu Rev Psychol, 2009, 60: 549-576.
24.	Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ, 2006, 332(7549): 1080.
25.	Bietenbeck A, Cervinski MA, Katayev A, et al. Understanding patient-based real-time quality control using simulation modeling. Clin Chem, 2020, 66(8): 1072-1083.
26.	谷鸿秋, 王俊峰, 章仲恒, 等. 临床预测模型: 模型的建立. 中国循证心血管医学杂志, 2019, 11(1): 14-16, 23.
27.	Rao SJ. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. J Am Stat Assoc, 2003, 98(461): 257-258.
28.	Heinze G, Wallisch C, Dunkler D. Variable selection - a review and recommendations for the practicing statistician. Biom J, 2018, 60(3): 431-449.
29.	Iasonos A, Schrag D, Raj GV, et al. How to build and interpret a nomogram for cancer prognosis. J Clin Oncol, 2008, 26(8): 1364-1370.
30.	Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med, 2019, 380(14): 1347-1358.
31.	Price WN. Big data and black-box medical algorithms. Sci Transl Med, 2018, 10(471): eaao5333.
32.	Salih A, Boscolo Galazzo I, Gkontra P, et al. Explainable artificial intelligence and cardiac imaging: toward more interpretable models. Circ Cardiovasc Imaging, 2023, 16(4): e014519.
33.	Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology, 2010, 21(1): 128-138.
34.	Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users' guides to the medical literature. JAMA, 2017, 318(14): 1377-1384.
35.	Pencina MJ, D'Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med, 2011, 30(1): 11-21.
36.	Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 2016, 352: i6.
37.	Fitzgerald M, Saville BR, Lewis RJ. Decision curve analysis. JAMA, 2015, 313(4): 409-410.
38.	Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016, 69: 245-247.
39.	Steyerberg EW. Validation in prediction research: the waste by data splitting. J Clin Epidemiol, 2018, 103: 131-133.
40.	Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ, 2009, 338: b605.
41.	Hutter F, Kotthoff L, Vanschoren J. Automated machine learning: methods, systems, challenges. Cham: Springer International Publishing, 2019.
42.	Debray TP, Vergouwe Y, Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol, 2015, 68(3): 279-289.
43.	Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med, 2016, 35(2): 214-226.
44.	Snell KIE, Archer L, Ensor J, et al. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. J Clin Epidemiol, 2021, 135: 79-89.
45.	Bonnett LJ, Snell KIE, Collins GS, et al. Guide to presenting clinical prediction models for use in clinical settings. BMJ, 2019, 365: l737.
46.	李笑丛, 王闯世, 郝军, 等. 临床预测模型校准与更新方法介绍及R软件实现. 中国循证医学杂志, 2023, 23(1): 112-119.
47.	曹煜隆, 单娇, 龚志忠, 等. 个体预后与诊断预测模型研究报告规范—TRIPOD声明解读. 中国循证医学杂志, 2020, 20(4): 492-496.
48.	Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
49.	Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med, 2019, 170(1): W1-W33.
50.	Liu Y, Chen PC, Krause J, et al. How to read articles that use machine learning: users' guides to the medical literature. JAMA, 2019, 322(18): 1806-1816.
51.	Faes L, Liu X, Wagner SK, et al. A clinician's guide to artificial intelligence: how to critically appraise machine learning studies. Transl Vis Sci Technol, 2020, 9(2): 7.
52.	Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open, 2021, 11(7): e048008.

1. Steyerberg EW, Moons KG, van der Windt DA, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med, 2013, 10(2): e1001381.
2. Moons KG, Royston P, Vergouwe Y, et al. Prognosis and prognostic research: what, why, and how. BMJ, 2009, 338: b375.
3. 谷鸿秋, 周支瑞, 章仲恒, 等. 临床预测模型: 基本概念、应用场景及研究思路. 中国循证心血管医学杂志, 2018, 10(12): 1454-1456, 1462.
4. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. Cham: Springer International Publishing, 2019.
5. Santos R, Coleman HG, Cairnduff V, et al. Clinical prediction models for pancreatic cancer in general and at-risk populations: a systematic review. Am J Gastroenterol, 2023, 118(1): 26-40.
6. Buttia C, Llanaj E, Raeisi-Dehkordi H, et al. Prognostic models in COVID-19 infection that predict severity: a systematic review. Eur J Epidemiol, 2023, 38(4): 355-372.
7. Kostopoulos G, Doundoulakis I, Toulis KA, et al. Prognostic models for heart failure in patients with type 2 diabetes: a systematic review and meta-analysis. Heart, 2023, 109(19): 1436-1442.
8. Pladet LCA, Barten JMM, Vernooij LM, et al. Prognostic models for mortality risk in patients requiring ECMO. Intensive Care Med, 2023, 49(2): 131-141.
9. Ohyama Y, Iwamura T, Hoshino T, et al. Prognostic models of quality of life after total knee replacement: a systematic review. Physiother Theory Pract, 2023.
10. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. BMJ, 2020: m1328.
11. Moons KG, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med, 2015, 162(1): W1-W73.
12. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med, 2006, 144(3): 201-209.
13. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Diabet Med, 2015, 32(2): 146-154.
14. Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med, 2014, 11(10): e1001744.
15. Pajouheshnia R, Groenwold RHH, Peelen LM, et al. When and how to use data from randomised trials to develop or validate prognostic models. BMJ, 2019, 365: l2154.
16. Harrell FE. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Cham: Springer International Publishing, 2015.
17. Hemingway H. Prognosis research: why is Dr. Lydgate still waiting? J Clin Epidemiol, 2006, 59(12): 1229-1238.
18. Clark TG, Bradburn MJ, Love SB, et al. Survival analysis part I: basic concepts and first analyses. Br J Cancer, 2003, 89(2): 232-238.
19. Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med, 2019, 38(7): 1276-1296.
20. van Smeden M, Moons KG, de Groot JA, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res, 2019, 28(8): 2455-2474.
21. Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ, 2020, 368: m441.
22. Little RJA, Rubin DB. Statistical analysis with missing data (3rd edition). Hoboken: Wiley, 2020.
23. Graham JW. Missing data analysis: making it work in the real world. Annu Rev Psychol, 2009, 60: 549-576.
24. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ, 2006, 332(7549): 1080.
25. Bietenbeck A, Cervinski MA, Katayev A, et al. Understanding patient-based real-time quality control using simulation modeling. Clin Chem, 2020, 66(8): 1072-1083.
26. 谷鸿秋, 王俊峰, 章仲恒, 等. 临床预测模型: 模型的建立. 中国循证心血管医学杂志, 2019, 11(1): 14-16, 23.
27. Rao SJ. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. J Am Stat Assoc, 2003, 98(461): 257-258.
28. Heinze G, Wallisch C, Dunkler D. Variable selection - a review and recommendations for the practicing statistician. Biom J, 2018, 60(3): 431-449.
29. Iasonos A, Schrag D, Raj GV, et al. How to build and interpret a nomogram for cancer prognosis. J Clin Oncol, 2008, 26(8): 1364-1370.
30. Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med, 2019, 380(14): 1347-1358.
31. Price WN. Big data and black-box medical algorithms. Sci Transl Med, 2018, 10(471): eaao5333.
32. Salih A, Boscolo Galazzo I, Gkontra P, et al. Explainable artificial intelligence and cardiac imaging: toward more interpretable models. Circ Cardiovasc Imaging, 2023, 16(4): e014519.
33. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology, 2010, 21(1): 128-138.
34. Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users' guides to the medical literature. JAMA, 2017, 318(14): 1377-1384.
35. Pencina MJ, D'Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med, 2011, 30(1): 11-21.
36. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 2016, 352: i6.
37. Fitzgerald M, Saville BR, Lewis RJ. Decision curve analysis. JAMA, 2015, 313(4): 409-410.
38. Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016, 69: 245-247.
39. Steyerberg EW. Validation in prediction research: the waste by data splitting. J Clin Epidemiol, 2018, 103: 131-133.
40. Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ, 2009, 338: b605.
41. Hutter F, Kotthoff L, Vanschoren J. Automated machine learning: methods, systems, challenges. Cham: Springer International Publishing, 2019.
42. Debray TP, Vergouwe Y, Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol, 2015, 68(3): 279-289.
43. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med, 2016, 35(2): 214-226.
44. Snell KIE, Archer L, Ensor J, et al. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. J Clin Epidemiol, 2021, 135: 79-89.
45. Bonnett LJ, Snell KIE, Collins GS, et al. Guide to presenting clinical prediction models for use in clinical settings. BMJ, 2019, 365: l737.
46. 李笑丛, 王闯世, 郝军, 等. 临床预测模型校准与更新方法介绍及R软件实现. 中国循证医学杂志, 2023, 23(1): 112-119.
47. 曹煜隆, 单娇, 龚志忠, 等. 个体预后与诊断预测模型研究报告规范—TRIPOD声明解读. 中国循证医学杂志, 2020, 20(4): 492-496.
48. Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med, 2019, 170(1): 51-58.
49. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med, 2019, 170(1): W1-W33.
50. Liu Y, Chen PC, Krause J, et al. How to read articles that use machine learning: users' guides to the medical literature. JAMA, 2019, 322(18): 1806-1816.
51. Faes L, Liu X, Wagner SK, et al. A clinician's guide to artificial intelligence: how to critically appraise machine learning studies. Transl Vis Sci Technol, 2020, 9(2): 7.
52. Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open, 2021, 11(7): e048008.

Chinese Journal of Evidence-Based Medicine

Methods and procedures of clinical predictive model

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content