An introduction to the calibration and update methods of clinical prediction models and its implementation by R software_Chinese Journal of Evidence-Based Medicine

Authors：

LI Xiaocong ,  WANG Chuangshi , HAO Jun , WANG Yang ,  LI Wei

Medical Research and Biometrics Center, Fuwai Hospital, National Center for Cardiovascular Disease, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 102300, P. R. China;

Corresponding author：

WANG Chuangshi, Email: wangchuagnshi@mrbc-nccd.com; LI Wei, Email: liwei@mrbc-nccd.com

Keywords：

Clinical prediction model; Model calibration; Model update; Software implementation

DOI：

10.7507/1672-2531.202209029

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The accuracy of the clinical prediction model determines its extrapolation and application value. When the prediction model is applied to a new setting, the differences between the new population and the initially modeled population in terms of study time, population characteristics, region, and other factors could lead to a reduction in its predictive performance. Calibrating or updating the prediction model with appropriate statistical methods is important to improve the accuracy of the prediction model in new populations. The model updating methods mainly include regression coefficients updating, meta-model updating and dynamic model updating. However, due to the limitations of meta-model updating and dynamic model updating in practical applications, the regression coefficient updating method is still the most common method in model updating. This paper introducd several types of model updating methods, the regression coefficients updating methods for two common clinical prediction models based on Logistic regression and Cox regression, and provide corresponding R codes for reference of researchers.

Citation： LI Xiaocong, WANG Chuangshi, HAO Jun, WANG Yang, LI Wei. An introduction to the calibration and update methods of clinical prediction models and its implementation by R software. Chinese Journal of Evidence-Based Medicine, 2023, 23(1): 112-119. doi: 10.7507/1672-2531.202209029 Copy

1.	Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594.
2.	谷鸿秋, 周支瑞, 章仲恒, 等. 临床预测模型: 基本概念, 应用场景及研究思路. 中国循证心血管医学杂志, 2018, 10(12): 1454-1456, 1462.
3.	谷鸿秋, 王俊峰, 章仲恒, 等. 临床预测模型: 模型的建立. 中国循证心血管医学杂志, 2019, 11(1): 14-16, 23.
4.	Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ, 2009, 338: b605.
5.	Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med, 1996, 15(4): 361-387.
6.	Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc, 2001, 8(4): 391-397.
7.	王俊峰, 章仲恒, 周支瑞, 等. 临床预测模型: 模型的验证. 中国循证心血管医学杂志, 2019, 11(2): 141-144.
8.	Janssen KJ, Moons KG, Kalkman CJ, et al. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol, 2008, 61(1): 76-86.
9.	Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol, 2008, 61(11): 1085-1094.
10.	Steyerberg EW, Borsboom GJ, van Houwelingen HC, et al. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med, 2004, 23(16): 2567-2586.
11.	Siregar S, Nieboer D, Versteegh MIM, et al. Methods for updating a risk prediction model for cardiac surgery: a statistical primer. Interact Cardiovasc Thorac Surg, 2019, 28(3): 333-338.
12.	Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: Ⅱ. External validation, model updating, and impact assessment. Heart, 2012, 98(9): 691-698.
13.	Cox DR. The regression analysis of binary sequences. J Royal Stat Society, 1958, 20: 215-232.
14.	Cox DR. Regression models and life-tables. J Royal Stat Society, 1972, 34(2): 187-220.
15.	Liu J, Hong Y, D'Agostino RB, et al. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese multi-provincial cohort study. JAMA, 2004, 291(21): 2591-2599.
16.	D'Agostino RB, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA, 2001, 286(2): 180-187.
17.	International Non-Hodgkin's Lymphoma Prognostic Factors Project. A predictive model for aggressive non-Hodgkin's lymphoma. N Engl J Med, 1993, 329(14): 987-994.
18.	van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Stat Med, 2000, 19(24): 3401-3415.
19.	Steyerberg E. Clinical prediction models: a practical approach to development, validation, and updating. Berlin: Springer, 2009.
20.	Su TL, Jaki T, Hickey GL, et al. A review of statistical updating methods for clinical prediction models. Stat Methods Med Res, 2018, 27(1): 185-197.
21.	Debray TP, Koffijberg H, Vergouwe Y, et al. Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med, 2012, 31(23): 2697-2712.
22.	Debray TP, Koffijberg H, Nieboer D, et al. Meta-analysis and aggregation of multiple published prediction models. Stat Med, 2014, 33(14): 2341-2362.
23.	Debray TP, Moons KG, Ahmed I, et al. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med, 2013, 32(18): 3158-3180.
24.	McCormick TH, Raftery AE, Madigan D, et al. Dynamic logistic regression and dynamic model averaging for binary classification. Biometrics, 2012, 68(1): 23-30.
25.	宋若齐, 吴疏桐, 王闯世. 医学研究中常见动态预测模型方法介绍. 中国循证医学杂志, 2022, 22(10): 1224-1232.
26.	Li B, Zhao Y, Cai W, et al. Validation and update of a multivariable prediction model for the identification and management of patients at risk for hepatocellular carcinoma. Clin Proteomics, 2021, 18(1): 21.
27.	Yang T, Xing H, Wang G, et al. A novel online calculator based on serum biomarkers to detect hepatocellular carcinoma among patients with hepatitis B. Clin Chem, 2019, 65(12): 1543-1553.
28.	Meech RW. An introduction to generalized linear models. J Royal Stat Society, 1992, 41: 217-218.
29.	Therneau TM, Grambsch PM. Modeling survival data: extending the cox model. Modeling Survival Data: Extending the Cox Model, 2013.
30.	Bradburn MJ, Clark TG, Love SB, et al. Survival analysis part II: multivariate data analysis-an introduction to concepts and methods. Br J Cancer, 2003, 89(3): 431-436.
31.	Andersen PK, Gill RD. Cox's regression model for counting processes: a large sample study. Ann Stat, 1982, 10: 1-41.
32.	陶立元, 刘小莉, 赵一鸣. 临床预测模型准确性影响因素及偏倚风险评估工具. 中华儿科杂志, 2021, 59(1): 2.

1. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ, 2015, 350: g7594.
2. 谷鸿秋, 周支瑞, 章仲恒, 等. 临床预测模型: 基本概念, 应用场景及研究思路. 中国循证心血管医学杂志, 2018, 10(12): 1454-1456, 1462.
3. 谷鸿秋, 王俊峰, 章仲恒, 等. 临床预测模型: 模型的建立. 中国循证心血管医学杂志, 2019, 11(1): 14-16, 23.
4. Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ, 2009, 338: b605.
5. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med, 1996, 15(4): 361-387.
6. Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc, 2001, 8(4): 391-397.
7. 王俊峰, 章仲恒, 周支瑞, 等. 临床预测模型: 模型的验证. 中国循证心血管医学杂志, 2019, 11(2): 141-144.
8. Janssen KJ, Moons KG, Kalkman CJ, et al. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol, 2008, 61(1): 76-86.
9. Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol, 2008, 61(11): 1085-1094.
10. Steyerberg EW, Borsboom GJ, van Houwelingen HC, et al. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med, 2004, 23(16): 2567-2586.
11. Siregar S, Nieboer D, Versteegh MIM, et al. Methods for updating a risk prediction model for cardiac surgery: a statistical primer. Interact Cardiovasc Thorac Surg, 2019, 28(3): 333-338.
12. Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: Ⅱ. External validation, model updating, and impact assessment. Heart, 2012, 98(9): 691-698.
13. Cox DR. The regression analysis of binary sequences. J Royal Stat Society, 1958, 20: 215-232.
14. Cox DR. Regression models and life-tables. J Royal Stat Society, 1972, 34(2): 187-220.
15. Liu J, Hong Y, D'Agostino RB, et al. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese multi-provincial cohort study. JAMA, 2004, 291(21): 2591-2599.
16. D'Agostino RB, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA, 2001, 286(2): 180-187.
17. International Non-Hodgkin's Lymphoma Prognostic Factors Project. A predictive model for aggressive non-Hodgkin's lymphoma. N Engl J Med, 1993, 329(14): 987-994.
18. van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Stat Med, 2000, 19(24): 3401-3415.
19. Steyerberg E. Clinical prediction models: a practical approach to development, validation, and updating. Berlin: Springer, 2009.
20. Su TL, Jaki T, Hickey GL, et al. A review of statistical updating methods for clinical prediction models. Stat Methods Med Res, 2018, 27(1): 185-197.
21. Debray TP, Koffijberg H, Vergouwe Y, et al. Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med, 2012, 31(23): 2697-2712.
22. Debray TP, Koffijberg H, Nieboer D, et al. Meta-analysis and aggregation of multiple published prediction models. Stat Med, 2014, 33(14): 2341-2362.
23. Debray TP, Moons KG, Ahmed I, et al. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med, 2013, 32(18): 3158-3180.
24. McCormick TH, Raftery AE, Madigan D, et al. Dynamic logistic regression and dynamic model averaging for binary classification. Biometrics, 2012, 68(1): 23-30.
25. 宋若齐, 吴疏桐, 王闯世. 医学研究中常见动态预测模型方法介绍. 中国循证医学杂志, 2022, 22(10): 1224-1232.
26. Li B, Zhao Y, Cai W, et al. Validation and update of a multivariable prediction model for the identification and management of patients at risk for hepatocellular carcinoma. Clin Proteomics, 2021, 18(1): 21.
27. Yang T, Xing H, Wang G, et al. A novel online calculator based on serum biomarkers to detect hepatocellular carcinoma among patients with hepatitis B. Clin Chem, 2019, 65(12): 1543-1553.
28. Meech RW. An introduction to generalized linear models. J Royal Stat Society, 1992, 41: 217-218.
29. Therneau TM, Grambsch PM. Modeling survival data: extending the cox model. Modeling Survival Data: Extending the Cox Model, 2013.
30. Bradburn MJ, Clark TG, Love SB, et al. Survival analysis part II: multivariate data analysis-an introduction to concepts and methods. Br J Cancer, 2003, 89(3): 431-436.
31. Andersen PK, Gill RD. Cox's regression model for counting processes: a large sample study. Ann Stat, 1982, 10: 1-41.
32. 陶立元, 刘小莉, 赵一鸣. 临床预测模型准确性影响因素及偏倚风险评估工具. 中华儿科杂志, 2021, 59(1): 2.

Chinese Journal of Evidence-Based Medicine

An introduction to the calibration and update methods of clinical prediction models and its implementation by R software

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content