Interpretation and application of the quality assessment tool (QATSM-RWS) for real-world studies in systematic reviews and meta-analyses_Chinese Journal of Evidence-Based Medicine

Authors：

ZHANG Qiang ¹ ,  LU Cuncun ² ,  ZHAO Wenxia ¹ , LIU Minghao ¹ , YAN Le ¹ , ZHANG Lihui ¹ , ZHAO Qing ¹ , LI Ling ¹ , WANG Siying ¹ , JIA Pan ¹ , SHANG Dongfang ¹ , FANG Jinhua ³

1. The First Affiliated Hospital of Henan University of Chinese Medicine, Zhengzhou 450000, P. R. China;
2. Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing 100700, P. R. China;
3. Jianshi Hospital of Traditional Chinese Medicine, Enshi 445300, P. R. China;

Corresponding author：

LU Cuncun, Email: cuncunlu2017@163.com; ZHAO Wenxia, Email: wenxia-zhao@163.com

Keywords：

Real-world study; Systematic review; Meta-analysis; Quality evaluation; Tool; MASLD

DOI：

10.7507/1672-2531.202406189

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Focusing on research quality is a crucial aspect of modern evidence-based medical practice, providing substantial evidence to underpin clinical decision-making. The increase in real-world studies in recent years has presented challenges, with varying quality stemming from issues such as data integrity and researchers’ expertise levels. Although systematic reviews and meta-analyses are essential references for clinical decisions, their reliability is contingent upon the quality of the primary studies. Making clinical decisions based on inadequate research poses inherent risks. With the lack of a specialized tool for evaluating the quality of real-world studies within systematic reviews and meta-analyses, the Gebrye team has introduced a new assessment tool - QATSM-RWS. Comprising 5 modules and 14 items, this tool aims to improve real-world research evaluation. This article aims to elaborate on the tool’s development process and content, taking using this tool to evaluate a published real-world study as an example and providing valuable guidance for domestic researchers utilizing this innovative tool.

1.	Liao X, Yao C, Zhang J, et al. Recent advancement in integrating artificial intelligence and information technology with real-world data for clinical decision-making in China: a scoping review. J Evid Based Med, 2023, 16(4): 534-546.
2.	Chen Z, Liu X, Hogan W, et al. Applications of artificial intelligence in drug development using real-world data. Drug Discov Today, 2021, 26(5): 1256-1264.
3.	Xu J, Wu W, Zhang X, et al. The use of real-world evidence for regulatory decisions in China. Clin Pharmacol Ther, 2024, 116(1): 82-95.
4.	卢存存, 陈子佳, 王志飞, 等. 基于真实世界数据的观察性因果推断研究新框架(目标试验模拟)及其在中医药领域中的应用展望. 协和医学杂志, 2024, 15(2): 422-428.
5.	Brunström M, Thomopoulos C, Carlberg B, et al. Methodological aspects of meta-analyses assessing the effect of blood pressure-lowering treatment on clinical outcomes. Hypertension, 2022, 79(3): 491-504.
6.	Büttner F, Winters M, Delahunt E, et al. Identifying the 'incredible'! Part 2: spot the difference - a rigorous risk of bias assessment can alter the main findings of a systematic review. Br J Sports Med, 2020, 54(13): 801-808.
7.	Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
8.	Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol, 2010, 25(9): 603-605.
9.	卢存存, 陈子佳, 张强, 等. 基于真实世界数据的目标试验模拟研究: 现状与展望. 中国循证医学杂志, 2023, 23(4): 492-496.
10.	Gebrye T, Mbada C, Hakimi Z, et al. Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey. Rheumatol Int, 2024, 44(7): 1275-1281.
11.	Gebrye T, Fatoye F, Mbada C, et al. A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies. Rheumatol Int, 2023, 43(9): 1573-1581.
12.	Ma YL, Ke JF, Wang JW, et al. Blood lactate levels are associated with an increased risk of metabolic dysfunction-associated fatty liver disease in type 2 diabetes: a real-world study. Front Endocrinol Lausanne, 2023, 14: 1133991.
13.	秦雪妮, 陈维生, 邵华, 等. 真实世界研究在医药领域的应用及研究方法. 药学进展, 2021, 45(7): 512-523.
14.	王瑞平, 李斌. 临床医学研究选题的规范和要点. 上海医药, 2023, 44(17): 47-50.
15.	Busse C, August E. How to write and publish a research paper for a peer-reviewed journal. J Cancer Educ, 2021, 36(5): 909-913.
16.	Bao H, Teplitskiy M. A simulation-based analysis of the impact of rhetorical citations in science. Nat Commun, 2024, 15(1): 431.
17.	Bozkurt S, Cahan EM, Seneviratne MG, et al. Reporting of demographic data and representativeness in machine learning models using electronic health records. J Am Med Inform Assoc, 2020, 27(12): 1878-1884.
18.	Desai RJ, Wang SV, Sreedhara SK, et al. Process guide for inferential studies using healthcare data from routine clinical practice to evaluate causal effects of drugs (PRINCIPLED): considerations from the FDA Sentinel Innovation Center. BMJ, 2024, 384: e076460.
19.	Stürmer T, Wang T, Golightly YM, et al. Methodological considerations when analysing and interpreting real-world data. Rheumatology Oxford, 2020, 59(1): 14-25.
20.	国家药监局药审中心. 药物真实世界研究设计与方案框架指导原则(试行). 2023.
21.	Fleurence RL, Kent S, Adamson B, et al. Assessing real-world data from electronic health records for health technology assessment: the SUITABILITY checklist: a good practices report of an ISPOR Task Force. Value Health, 2024, 27(6): 692-701.
22.	翟静波, 郑文科, 王辉, 等. 真实世界研究样本量估计的统计学考虑. 世界中医药, 2019, 14(12): 3123-3126.
23.	赵瑞霞, 邵明义, 符宇, 等. 基于真实世界中医电子病历数据的原发性肝癌疗效评价难点及解决策略. 中医杂志, 2019, 60(23): 2009-2012.
24.	贾玉龙, 姚明宏, 徐嘉悦, 等. 真实世界研究中 Zelen 设计应用探讨. 中国循证医学杂志, 2022, 22(5): 615-620.
25.	张强, 蒙萍, 单爱莲. 关于药物临床试验方案中纳入、排除标准的若干思考. 中国临床药理学杂志, 2017, 33(2): 99-101.
26.	彭晓霞, 舒啸尘, 谭婧, 等. 基于真实世界数据评价治疗结局的观察性研究设计技术规范. 中国循证医学杂志, 2019, 19(7): 779-786.
27.	史梦龙, 张晨瑶, 吴晓蕾, 等. “以患者为中心药物研发”中患者体验数据的收集技术要点. 中国循证医学杂志, 2024, 24(4): 478-483.
28.	Kottner J, Beaton D, Clarke M, et al. Core outcome set developers should consider and specify the level of granularity of outcome domains. J Clin Epidemiol, 2024, 169: 111307.
29.	王琳琳, 赵瑞霞, 符宇, 等. 真实世界研究中随访面临的问题及解决策略. 空军军医大学学报, 2022, 43(9): 1051-1054.
30.	高培, 王杨, 罗剑锋, 等. 基于真实世界数据评价治疗结局研究的统计分析技术规范. 中国循证医学杂志, 2019, 19(7): 787-793.
31.	晋菲斐, 阎小妍, 董冲亚, 等. 真实世界研究方案设计要点及统计分析解析. 中国卒中杂志, 2022, 17(12): 1294-1298.
32.	张颖, 王丽琼, 费宇彤, 等. 统计分析计划(SAP)报告指南的解读及其对中医药临床研究的启示. 中医杂志, 2019, 60(5): 385-390.
33.	Castelo-Branco L, Pellat A, Martins-Branco D, et al. ESMO guidance for reporting oncology real-world evidence (GROW). Ann Oncol, 2023, 34(12): 1097-1112.
34.	金鑫瑶, 郑文科, 张俊华, 等. 推进真实世界研究的透明化. 世界中医药, 2019, 14(12): 3106-3110.
35.	李雨芯, 熊俊, 张政, 等. 中药系统评价/Meta分析报告规范(PRISMA-CHM)解读. 中国循证医学杂志, 2023, 23(11): 1351-1359.
36.	Akl EA, Hakoum M, Khamis A, et al. A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. J Clin Epidemiol, 2022, 149: 236-243.
37.	Siena LM, Papamanolis L, Siebert MJ, et al. Industry involvement and transparency in the most cited clinical trials, 2019-2022. JAMA Netw Open, 2023, 6(11): e2343425.
38.	DeVos E, Simon EL, Aluisio A. Funding sources for research: a research primer for low- and middle-income countries. Afr J Emerg Med, 2020, 10(Suppl 2): S130-S134.
39.	Jansen MS, Dekkers OM, le Cessie S, et al. Real-world evidence to inform regulatory decision making: a scoping review. Clin Pharmacol Ther, 2024, 115(6): 1269-1276.
40.	Lu C, Ke L, Zhang Q, et al. Quality of systematic reviews with meta-analyses of resveratrol: a methodological systematic review. Phytother Res, 2024, 38(1): 11-21.
41.	Fatoye F, Smith P, Gebrye T, et al. Real-world persistence and adherence with oral bisphosphonates for osteoporosis: a systematic review. BMJ Open, 2019, 9(4): e027049.
42.	Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.

1. Liao X, Yao C, Zhang J, et al. Recent advancement in integrating artificial intelligence and information technology with real-world data for clinical decision-making in China: a scoping review. J Evid Based Med, 2023, 16(4): 534-546.
2. Chen Z, Liu X, Hogan W, et al. Applications of artificial intelligence in drug development using real-world data. Drug Discov Today, 2021, 26(5): 1256-1264.
3. Xu J, Wu W, Zhang X, et al. The use of real-world evidence for regulatory decisions in China. Clin Pharmacol Ther, 2024, 116(1): 82-95.
4. 卢存存, 陈子佳, 王志飞, 等. 基于真实世界数据的观察性因果推断研究新框架(目标试验模拟)及其在中医药领域中的应用展望. 协和医学杂志, 2024, 15(2): 422-428.
5. Brunström M, Thomopoulos C, Carlberg B, et al. Methodological aspects of meta-analyses assessing the effect of blood pressure-lowering treatment on clinical outcomes. Hypertension, 2022, 79(3): 491-504.
6. Büttner F, Winters M, Delahunt E, et al. Identifying the 'incredible'! Part 2: spot the difference - a rigorous risk of bias assessment can alter the main findings of a systematic review. Br J Sports Med, 2020, 54(13): 801-808.
7. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
8. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol, 2010, 25(9): 603-605.
9. 卢存存, 陈子佳, 张强, 等. 基于真实世界数据的目标试验模拟研究: 现状与展望. 中国循证医学杂志, 2023, 23(4): 492-496.
10. Gebrye T, Mbada C, Hakimi Z, et al. Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey. Rheumatol Int, 2024, 44(7): 1275-1281.
11. Gebrye T, Fatoye F, Mbada C, et al. A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies. Rheumatol Int, 2023, 43(9): 1573-1581.
12. Ma YL, Ke JF, Wang JW, et al. Blood lactate levels are associated with an increased risk of metabolic dysfunction-associated fatty liver disease in type 2 diabetes: a real-world study. Front Endocrinol Lausanne, 2023, 14: 1133991.
13. 秦雪妮, 陈维生, 邵华, 等. 真实世界研究在医药领域的应用及研究方法. 药学进展, 2021, 45(7): 512-523.
14. 王瑞平, 李斌. 临床医学研究选题的规范和要点. 上海医药, 2023, 44(17): 47-50.
15. Busse C, August E. How to write and publish a research paper for a peer-reviewed journal. J Cancer Educ, 2021, 36(5): 909-913.
16. Bao H, Teplitskiy M. A simulation-based analysis of the impact of rhetorical citations in science. Nat Commun, 2024, 15(1): 431.
17. Bozkurt S, Cahan EM, Seneviratne MG, et al. Reporting of demographic data and representativeness in machine learning models using electronic health records. J Am Med Inform Assoc, 2020, 27(12): 1878-1884.
18. Desai RJ, Wang SV, Sreedhara SK, et al. Process guide for inferential studies using healthcare data from routine clinical practice to evaluate causal effects of drugs (PRINCIPLED): considerations from the FDA Sentinel Innovation Center. BMJ, 2024, 384: e076460.
19. Stürmer T, Wang T, Golightly YM, et al. Methodological considerations when analysing and interpreting real-world data. Rheumatology Oxford, 2020, 59(1): 14-25.
20. 国家药监局药审中心. 药物真实世界研究设计与方案框架指导原则(试行). 2023.
21. Fleurence RL, Kent S, Adamson B, et al. Assessing real-world data from electronic health records for health technology assessment: the SUITABILITY checklist: a good practices report of an ISPOR Task Force. Value Health, 2024, 27(6): 692-701.
22. 翟静波, 郑文科, 王辉, 等. 真实世界研究样本量估计的统计学考虑. 世界中医药, 2019, 14(12): 3123-3126.
23. 赵瑞霞, 邵明义, 符宇, 等. 基于真实世界中医电子病历数据的原发性肝癌疗效评价难点及解决策略. 中医杂志, 2019, 60(23): 2009-2012.
24. 贾玉龙, 姚明宏, 徐嘉悦, 等. 真实世界研究中 Zelen 设计应用探讨. 中国循证医学杂志, 2022, 22(5): 615-620.
25. 张强, 蒙萍, 单爱莲. 关于药物临床试验方案中纳入、排除标准的若干思考. 中国临床药理学杂志, 2017, 33(2): 99-101.
26. 彭晓霞, 舒啸尘, 谭婧, 等. 基于真实世界数据评价治疗结局的观察性研究设计技术规范. 中国循证医学杂志, 2019, 19(7): 779-786.
27. 史梦龙, 张晨瑶, 吴晓蕾, 等. “以患者为中心药物研发”中患者体验数据的收集技术要点. 中国循证医学杂志, 2024, 24(4): 478-483.
28. Kottner J, Beaton D, Clarke M, et al. Core outcome set developers should consider and specify the level of granularity of outcome domains. J Clin Epidemiol, 2024, 169: 111307.
29. 王琳琳, 赵瑞霞, 符宇, 等. 真实世界研究中随访面临的问题及解决策略. 空军军医大学学报, 2022, 43(9): 1051-1054.
30. 高培, 王杨, 罗剑锋, 等. 基于真实世界数据评价治疗结局研究的统计分析技术规范. 中国循证医学杂志, 2019, 19(7): 787-793.
31. 晋菲斐, 阎小妍, 董冲亚, 等. 真实世界研究方案设计要点及统计分析解析. 中国卒中杂志, 2022, 17(12): 1294-1298.
32. 张颖, 王丽琼, 费宇彤, 等. 统计分析计划(SAP)报告指南的解读及其对中医药临床研究的启示. 中医杂志, 2019, 60(5): 385-390.
33. Castelo-Branco L, Pellat A, Martins-Branco D, et al. ESMO guidance for reporting oncology real-world evidence (GROW). Ann Oncol, 2023, 34(12): 1097-1112.
34. 金鑫瑶, 郑文科, 张俊华, 等. 推进真实世界研究的透明化. 世界中医药, 2019, 14(12): 3106-3110.
35. 李雨芯, 熊俊, 张政, 等. 中药系统评价/Meta分析报告规范(PRISMA-CHM)解读. 中国循证医学杂志, 2023, 23(11): 1351-1359.
36. Akl EA, Hakoum M, Khamis A, et al. A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. J Clin Epidemiol, 2022, 149: 236-243.
37. Siena LM, Papamanolis L, Siebert MJ, et al. Industry involvement and transparency in the most cited clinical trials, 2019-2022. JAMA Netw Open, 2023, 6(11): e2343425.
38. DeVos E, Simon EL, Aluisio A. Funding sources for research: a research primer for low- and middle-income countries. Afr J Emerg Med, 2020, 10(Suppl 2): S130-S134.
39. Jansen MS, Dekkers OM, le Cessie S, et al. Real-world evidence to inform regulatory decision making: a scoping review. Clin Pharmacol Ther, 2024, 115(6): 1269-1276.
40. Lu C, Ke L, Zhang Q, et al. Quality of systematic reviews with meta-analyses of resveratrol: a methodological systematic review. Phytother Res, 2024, 38(1): 11-21.
41. Fatoye F, Smith P, Gebrye T, et al. Real-world persistence and adherence with oral bisphosphonates for osteoporosis: a systematic review. BMJ Open, 2019, 9(4): e027049.
42. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.

Chinese Journal of Evidence-Based Medicine

Latest ArticlesInterpretation and application of the quality assessment tool (QATSM-RWS) for real-world studies in systematic reviews and meta-analyses

Abstract Full text Figures/Tables Video References Cited by

Format

Content