Utilization of the evidence from studies with no events in meta-analyses of adverse events: An empirical investigation (Chinese translation)_Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Authors：

 XU Chang ^1,2 , ZHOU Xiaoqin ² , Zorzela Liliane ³ , JU Ke ⁴ , Furuya-Kanamori Luis ⁵ , LIN Lifeng ⁶ , LU Cuncun ^7,8 , Musa Omran A.H. ¹ , Vohra Sunita ^3,9

1. Department of Population Medicine, College of Medicine, Qatar University, Doha, Qatar;
2. Department of Clinical Research Management, West China Hospital, Sichuan University, Chengdu, 610041, P.R.China;
3. Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada;
4. West China School of Public Health and West China Fourth Hospital, Sichuan University, Chengdu, 610041, P.R.China;
5. UQ Centre for Clinical Research, Faculty of Medicine, University of Queensland, Brisbane, Australia;
6. Department of Statistics, Florida State University, Tallahassee, Florida, USA;
7. Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, 730000, P.R.China;
8. Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, 100070, P.R.China;
9. Department of Psychiatry, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada;

Corresponding author：

XU Chang, Email: xuchang2016@runbox.com

Keywords：

Systematic review; meta-analysis; adverse events; zero-events studies

DOI：

10.7507/1007-4848.202105085

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective Zero-events studies frequently occur in systematic reviews of adverse events, which consist of an important source of evidence. We aimed to examine how evidence of zero-events studies was utilized in the meta-analyses of systematic reviews of adverse events.Methods We conducted a survey of systematic reviews published in two periods: January 1, 2015 to January 1, 2020 and January 1, 2008, to April 25, 2011. Databases were searched for systematic reviews that conducted at least one meta-analysis of any healthcare intervention and used adverse events as the exclusive outcome. An adverse event was defined as any untoward medical occurrence in a patient or subject in healthcare practice. We summarized the frequency of occurrence of zero-events studies in eligible systematic reviews and how these studies were dealt with in the meta-analyses of these systematic reviews.Results We included 640 eligible systematic reviews. There were 406 (63.45%) systematic reviews involving zero-events studies in their meta-analyses, among which 389 (95.11%) involved single-arm-zero-events studies and 223 (54.93%) involved double-arm-zero-events studies. The majority (98.71%) of these systematic reviews incorporated single-arm-zero-events studies into the meta-analyses. On the other hand, the majority (76.23%) of them excluded double-arm-zero-events studies from the meta-analyses, of which the majority (87.06%) did not discuss the potential impact of excluding such studies. Systematic reviews published at present (2015-2020) tended to incorporate zero-events studies in meta-analyses than those published in the past (2008-2011), but the difference was not significant [proportion difference=–0.09, 95%CI (–0.21, 0.03), P=0.12].Conclusion Systematic review authors routinely treated studies with zero-events in both arms as "non-informative" carriers and excluded them from their reviews. Whether studies with no events are "informative" or not, largely depends on the methods and assumptions applied, thus sensitivity analyses using different methods should be considered in future meta-analyses.

Citation： XU Chang, ZHOU Xiaoqin, Zorzela Liliane, JU Ke, Furuya-Kanamori Luis, LIN Lifeng, LU Cuncun, Musa Omran A.H., Vohra Sunita. Utilization of the evidence from studies with no events in meta-analyses of adverse events: An empirical investigation (Chinese translation). Chinese Journal of Clinical Thoracic and Cardiovascular Surgery, 2021, 28(12): 1494-1502. doi: 10.7507/1007-4848.202105085 Copy

1.	Egger M, Smith GD, Phillips AN. Meta-analysis: Principles and procedures. BMJ, 1997, 315(7121): 1533-1537.
2.	Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care: Meta-analysis in context. Chapter 15. Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. BMJ Books, 2001: 285-312.
3.	DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials, 1986, 7(3): 177.
4.	Bhaumik DK, Amatya A, Normand SL, et al. Meta-analysis of rare binary adverse event data. J Am Stat Assoc, 2012, 107(498): 555-567.
5.	Efthimiou O. Practical guide to the meta-analysis of rare events. Evid Based Ment Health, 2018, 21(2): 72-76.
6.	Kuss O. Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless. Stat Med, 2015, 34(7): 1097-1116.
7.	Yusuf S, Peto R, Lewis J, et al. Beta blockade during and after myocardial infarction: An overview of the randomized trials. Prog Cardiovasc Dis, 1985, 27(5): 335-371.
8.	Cox, DR. The continuity correction. Biometrika, 1970, 57(1): 217-219.
9.	Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst, 1959, 22(4): 719-48.
10.	Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Stat Med, 1991, 10(11): 1665-1677.
11.	Böhning D, Mylona K, Kimber A. Meta-analysis of clinical trials with rare events. Biom J, 2015, 57(4): 633-648.
12.	Xu C, Li L, Lin L, et al. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol, 2020, 123: 91-99.
13.	Friedrich JO, Adhikari NK, Beyene J. Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Med Res Methodol, 2007, 7: 5.
14.	Platt RW, Leroux BG, Breslow N. Generalized linear mixed models for meta-analysis. Stat Med, 1999, 18(6): 643-654.
15.	Simmonds MC, Higgins JP. A general framework for the use of logistic regression models in meta-analysis. Stat Methods Med Res, 2016, 25(6): 2858-2877.
16.	Jackson D, Law M, Stijnen T, et al. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat Med, 2018, 37(7): 1059-1085.
17.	Chu H, Nie L, Chen Y, et al. Bivariate random effects models for meta-analysis of comparative studies with binary outcomes: Methods for the absolute risk difference and relative risk. Stat Methods Med Res, 2012, 21(6): 621-633.
18.	Mathes T, Kuss O. Beta-binomial models for meta-analysis with binary outcomes: Variations, extensions, and additional insights from econometrics. Rese Meth Med Health Sci, 2021, 2(2): 82-89.
19.	Böhning D, Sangnawakij P. The identity of two meta-analytic likelihoods and the ignorability of double-zero studies. Biostatistics. 2020: kxaa004.
20.	Zorzela L, Golder S, Liu Y, et al. Quality of reporting in systematic reviews of adverse events: Systematic review. BMJ, 2014, 348: f7668.
21.	Xu C, Furuya-Kanamori L, Zorzela L, et al. A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies. J Clin Epidemiol, 2021, 135: 70-78.
22.	Bougioukas KI, Liakos A, Tsapas A, et al. Preferred reporting items for overviews of systematic reviews including harms checklist: A pilot tool to be used for balanced reporting of benefits and harms. J Clin Epidemiol, 2018, 93: 9-24.
23.	FDA. What is a serious adverse event? https://www.fda.gov/safety/reporting-serious-problems-fda/what-serious-adverse-event. Accessed on 2020-10-21.
24.	Zorzela L, Loke YK, Ioannidis JP, et al. PRISMA harms checklist: Improving harms reporting in systematic reviews. BMJ, 2016, 352: i157.
25.	Xu C, Liu Y, Jia PL, et al. The methodological quality of dose-response meta-analyses needed substantial improvement: A cross-sectional survey and proposed recommendations. J Clin Epidemiol, 2019, 107: 1-11.
26.	Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chichester (UK): John Wiley & Sons, 2019.
27.	Shan G, Gerstenberger S. Fisher's exact approach for post hoc analysis of a chi-squared test. PLoS One, 2017, 12(12): e0188709.
28.	Zhou Y, Zhu B, Lin L, et al. Protocols for meta-analysis of intervention safety seldom specified methods to deal with rare events. J Clin Epidemiol, 2020, 128: 109-117.
29.	Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med, 2004, 23(9): 1351-1375.
30.	Rücker G, Schumacher M. Simpson's paradox visualized: The example of the rosiglitazone meta-analysis. BMC Med Res Methodol, 2008, 8: 34.
31.	Berstock J, Beswick A. Importance of contacting authors for data on adverse events when compiling systematic reviews. BMJ, 2014, 348: g1394.
32.	Saini P, Loke YK, Gamble C, et al. Selective reporting bias of harm outcomes within studies: Findings from a cohort of systematic reviews. BMJ, 2014, 349: g6501.
33.	Xie MG, Kolassa J, Liu DG, et al. Does an observed zero-total-event study contain information for inference of odds ratio in meta-analysis? Statistics and its Interface, 2018, 11: 327-337.
34.	Cai T, Parast L, Ryan L. Meta-analysis for rare events. Stat Med, 2010, 29(20): 2078-2089.
35.	Xiao M, Lin L, Hodges JS, et al. Double-zero-event studies matter: A re-evaluation of physical distancing, face masks, and eye protection for preventing person-to-person transmission of COVID-19 and its policy impact. J Clin Epidemiol, 2021: S0895-4356(21)00032-9.
36.	Al Amer FM, Thompson CG, Lin L. Bayesian methods for meta-analyses of binary outcomes: Implementations, examples, and impact of priors. Int J Environ Res Public Health, 2021, 18(7): 3492.
37.	Klingenberg B. A new and improved confidence interval for the Mantel-Haenszel risk difference. Stat Med, 2014, 33(17): 2968-2983.
38.	Liu D, Liu RY, Xie M. Exact meta-analysis approach for discrete data and its application to 2×2 tables with rare events. J Am Stat Assoc, 2014, 109(508): 1450-1465.
39.	Bradburn MJ, Deeks JJ, Berlin JA, et al. Much ado about nothing: A comparison of the performance of meta-analytical methods with rare events. Stat Med, 2007, 26(1): 53-77.
40.	Jia P, Lin L, Kwong JSW, et al. Many meta-analyses of rare events in the Cochrane Database of Systematic Reviews were underpowered. J Clin Epidemiol, 2020: S0895-4356(20)31188-4.
41.	Jackson D, Turner R. Power analysis for random-effects meta-analysis. Res Synth Methods, 2017, 8(3): 290-302.
42.	Zeger SL, Liang KY, Albert PS. Models for longitudinal data: A generalized estimating equation approach. Biometrics, 1988, 44(4): 1049-1060.
43.	Ju K, Lin L, Chu H, et al. Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: Towards meta-analysis of binary outcome with sparse data. BMC Med Res Methodol, 2020, 20(1): 152.
44.	Altman DG, Bland JM. Missing data. BMJ, 2007, 334(7590): 424.
45.	Kahale LA, Khamis AM, Diab B, et al. Potential impact of missing outcome data on treatment effects in systematic reviews: Imputation study. BMJ, 2020, 370: m2898.

1. Egger M, Smith GD, Phillips AN. Meta-analysis: Principles and procedures. BMJ, 1997, 315(7121): 1533-1537.
2. Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care: Meta-analysis in context. Chapter 15. Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. BMJ Books, 2001: 285-312.
3. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials, 1986, 7(3): 177.
4. Bhaumik DK, Amatya A, Normand SL, et al. Meta-analysis of rare binary adverse event data. J Am Stat Assoc, 2012, 107(498): 555-567.
5. Efthimiou O. Practical guide to the meta-analysis of rare events. Evid Based Ment Health, 2018, 21(2): 72-76.
6. Kuss O. Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless. Stat Med, 2015, 34(7): 1097-1116.
7. Yusuf S, Peto R, Lewis J, et al. Beta blockade during and after myocardial infarction: An overview of the randomized trials. Prog Cardiovasc Dis, 1985, 27(5): 335-371.
8. Cox, DR. The continuity correction. Biometrika, 1970, 57(1): 217-219.
9. Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst, 1959, 22(4): 719-48.
10. Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Stat Med, 1991, 10(11): 1665-1677.
11. Böhning D, Mylona K, Kimber A. Meta-analysis of clinical trials with rare events. Biom J, 2015, 57(4): 633-648.
12. Xu C, Li L, Lin L, et al. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol, 2020, 123: 91-99.
13. Friedrich JO, Adhikari NK, Beyene J. Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Med Res Methodol, 2007, 7: 5.
14. Platt RW, Leroux BG, Breslow N. Generalized linear mixed models for meta-analysis. Stat Med, 1999, 18(6): 643-654.
15. Simmonds MC, Higgins JP. A general framework for the use of logistic regression models in meta-analysis. Stat Methods Med Res, 2016, 25(6): 2858-2877.
16. Jackson D, Law M, Stijnen T, et al. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat Med, 2018, 37(7): 1059-1085.
17. Chu H, Nie L, Chen Y, et al. Bivariate random effects models for meta-analysis of comparative studies with binary outcomes: Methods for the absolute risk difference and relative risk. Stat Methods Med Res, 2012, 21(6): 621-633.
18. Mathes T, Kuss O. Beta-binomial models for meta-analysis with binary outcomes: Variations, extensions, and additional insights from econometrics. Rese Meth Med Health Sci, 2021, 2(2): 82-89.
19. Böhning D, Sangnawakij P. The identity of two meta-analytic likelihoods and the ignorability of double-zero studies. Biostatistics. 2020: kxaa004.
20. Zorzela L, Golder S, Liu Y, et al. Quality of reporting in systematic reviews of adverse events: Systematic review. BMJ, 2014, 348: f7668.
21. Xu C, Furuya-Kanamori L, Zorzela L, et al. A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies. J Clin Epidemiol, 2021, 135: 70-78.
22. Bougioukas KI, Liakos A, Tsapas A, et al. Preferred reporting items for overviews of systematic reviews including harms checklist: A pilot tool to be used for balanced reporting of benefits and harms. J Clin Epidemiol, 2018, 93: 9-24.
23. FDA. What is a serious adverse event? https://www.fda.gov/safety/reporting-serious-problems-fda/what-serious-adverse-event. Accessed on 2020-10-21.
24. Zorzela L, Loke YK, Ioannidis JP, et al. PRISMA harms checklist: Improving harms reporting in systematic reviews. BMJ, 2016, 352: i157.
25. Xu C, Liu Y, Jia PL, et al. The methodological quality of dose-response meta-analyses needed substantial improvement: A cross-sectional survey and proposed recommendations. J Clin Epidemiol, 2019, 107: 1-11.
26. Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chichester (UK): John Wiley & Sons, 2019.
27. Shan G, Gerstenberger S. Fisher's exact approach for post hoc analysis of a chi-squared test. PLoS One, 2017, 12(12): e0188709.
28. Zhou Y, Zhu B, Lin L, et al. Protocols for meta-analysis of intervention safety seldom specified methods to deal with rare events. J Clin Epidemiol, 2020, 128: 109-117.
29. Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med, 2004, 23(9): 1351-1375.
30. Rücker G, Schumacher M. Simpson's paradox visualized: The example of the rosiglitazone meta-analysis. BMC Med Res Methodol, 2008, 8: 34.
31. Berstock J, Beswick A. Importance of contacting authors for data on adverse events when compiling systematic reviews. BMJ, 2014, 348: g1394.
32. Saini P, Loke YK, Gamble C, et al. Selective reporting bias of harm outcomes within studies: Findings from a cohort of systematic reviews. BMJ, 2014, 349: g6501.
33. Xie MG, Kolassa J, Liu DG, et al. Does an observed zero-total-event study contain information for inference of odds ratio in meta-analysis? Statistics and its Interface, 2018, 11: 327-337.
34. Cai T, Parast L, Ryan L. Meta-analysis for rare events. Stat Med, 2010, 29(20): 2078-2089.
35. Xiao M, Lin L, Hodges JS, et al. Double-zero-event studies matter: A re-evaluation of physical distancing, face masks, and eye protection for preventing person-to-person transmission of COVID-19 and its policy impact. J Clin Epidemiol, 2021: S0895-4356(21)00032-9.
36. Al Amer FM, Thompson CG, Lin L. Bayesian methods for meta-analyses of binary outcomes: Implementations, examples, and impact of priors. Int J Environ Res Public Health, 2021, 18(7): 3492.
37. Klingenberg B. A new and improved confidence interval for the Mantel-Haenszel risk difference. Stat Med, 2014, 33(17): 2968-2983.
38. Liu D, Liu RY, Xie M. Exact meta-analysis approach for discrete data and its application to 2×2 tables with rare events. J Am Stat Assoc, 2014, 109(508): 1450-1465.
39. Bradburn MJ, Deeks JJ, Berlin JA, et al. Much ado about nothing: A comparison of the performance of meta-analytical methods with rare events. Stat Med, 2007, 26(1): 53-77.
40. Jia P, Lin L, Kwong JSW, et al. Many meta-analyses of rare events in the Cochrane Database of Systematic Reviews were underpowered. J Clin Epidemiol, 2020: S0895-4356(20)31188-4.
41. Jackson D, Turner R. Power analysis for random-effects meta-analysis. Res Synth Methods, 2017, 8(3): 290-302.
42. Zeger SL, Liang KY, Albert PS. Models for longitudinal data: A generalized estimating equation approach. Biometrics, 1988, 44(4): 1049-1060.
43. Ju K, Lin L, Chu H, et al. Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: Towards meta-analysis of binary outcome with sparse data. BMC Med Res Methodol, 2020, 20(1): 152.
44. Altman DG, Bland JM. Missing data. BMJ, 2007, 334(7590): 424.
45. Kahale LA, Khamis AM, Diab B, et al. Potential impact of missing outcome data on treatment effects in systematic reviews: Imputation study. BMJ, 2020, 370: m2898.

Previous Article
Multidisciplinary management of acute type A aortic dissection in late pregnancy
Next Article
Research progress on risk factors for acute aortic dissection complicated with acute lung injury

Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Utilization of the evidence from studies with no events in meta-analyses of adverse events: An empirical investigation (Chinese translation)

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content