Method of using RE-AIM framework to develop implementation outcome indicators_Chinese Journal of Evidence-Based Medicine

Authors：

LV Sensen ¹ , CAI Yiyuan ² , HE Wenjun ¹ , CAI Yefeng ³ , NI Xiaojia ³ , LUO Huanyuan ¹ , ZHONG Dongmei ¹ , LIU Siyuan ¹ , ZHAO Qing ¹ , ZHANG Lanping ¹ , LI Jiaqi ¹ , CHEN Xiaoshan ¹ , XIE Yunyun ¹ , LI Chunping ¹ , HUANG Wanqing ¹ ,  XU Dong ^1,4,5

1. Acacia Lab for Implementation Science, School of Health Management, Southern Medical University, Guangzhou 510515, P. R. China;
2. School of Public Health, Guizhou Medical University, Guiyang 550000, P. R. China;
3. Provincial Hospital of Chinese Medicine, Guangzhou 510120, P. R. China;
4. Center for World Health Organization Studies and Department of Health Management, School of Health Management of Southern Medical University, Guangzhou 510515, P. R. China;
5. Institute for Global Health, Dermatology Hospital of Southern Medical University, Guangzhou 510515, P. R. China;

Corresponding author：

XU Dong, Email: romanxu@i.smu.edu.cn

Keywords：

Implementation science; Implementation research; Evidence-based practice; RE-AIM framework; Guideline

DOI：

10.7507/1672-2531.202211159

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The RE-AIM framework is the most commonly used framework for implementation research outcome evaluation, with the original paper cited 2 800 times. However, in the specific operationalization process of the framework, there are often situations of misuse or overuse of various dimensions and indicators. This article provides a detailed introduction to the main content, application scenarios, and characteristics of the RE-AIM framework. Taking the implementation research of Evidence-based Practice Guidelines for Integrated Traditional Chinese and Western Medicine Treatment of Stroke as an example, it introduces how to use the silent & anonymous feedback technology, which combines the improved Delphi method and nominal group technique, to construct an implementation outcome indicator evaluation system to achieve the operability of the RE-AIM framework. This will provide a reference for other researchers to use implementation science theories, models, and frameworks to develop implementation plans or evaluate implementation outcomes.

Citation： LV Sensen, CAI Yiyuan, HE Wenjun, CAI Yefeng, NI Xiaojia, LUO Huanyuan, ZHONG Dongmei, LIU Siyuan, ZHAO Qing, ZHANG Lanping, LI Jiaqi, CHEN Xiaoshan, XIE Yunyun, LI Chunping, HUANG Wanqing, XU Dong. Method of using RE-AIM framework to develop implementation outcome indicators. Chinese Journal of Evidence-Based Medicine, 2023, 23(6): 695-701. doi: 10.7507/1672-2531.202211159 Copy

1.	Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med, 2011, 104(12): 510-520.
2.	Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform, 2000, 9(1): 65-70.
3.	陈文嘉, 徐东, 李慧, 等. 实施科学理论的分类与介绍. 中国循证医学杂志, 2020, 20(8): 986-992.
4.	屈智勇, 郭帅, 张维军, 等. 实施科学对我国心理健康服务体系建设的启示. 北京师范大学学报(社会科学版), 2017, (2): 29-36.
5.	Nutley S, Walter I, Davies HTO. From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation, 2003, 9(2): 125-148.
6.	Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatry Res, 2020, 283: 112461.
7.	Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci, 2015, 10: 53.
8.	Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health, 1999, 89(9): 1322-1327.
9.	Proctor EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health, 2009, 36(1): 24-34.
10.	Green LW, Kreuter MW. Health promotion planning: an educational and ecological approach (3rd edition). Mountain View, CA: Mayfield Publishing, 1999.
11.	Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2019, 7: 64.
12.	Tabak RG, Khoong EC, Chambers DA, et al. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med, 2012, 43(3): 337-350.
13.	Stange KC. Commentary: RE-AIM Planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2020, 8: 245.
14.	King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health, 2010, 100(11): 2076-2084.
15.	中华中医药学会脑病分会, 广东省中医药学会脑病专业委员会, 广东省中西医结合学会卒中专业委员会. 中西医结合脑卒中循证实践指南(2019). 中国循证医学杂志, 2020, 20(8): 901-912.
16.	Glasgow RE, Estabrooks PA, Ory MG. Characterizing evolving frameworks: issues from Esmail et al. (2020) review. Implement Sci, 2020, 15(1): 53.
17.	Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol, 2004, 23(5): 443-451.
18.	娄晓民. RE-AIM模型. 余金明, 姜庆五, 主编. 现代健康教育学. 上海: 复旦大学出版社, 2019: 120-123.
19.	Holtrop JS, Estabrooks PA, Gaglio B, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci, 2021, 5(1): e126.
20.	田文华. 基于RE-AIM模型的城乡居民大病保险模式评估. 复旦学报(社会科学版), 2021, 63(1): 152-160.
21.	Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health, 2013, 103(6): e38-e46.
22.	Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf, 2008, 34(4): 228-243.
23.	Liles EG, Schneider JL, Feldstein AC, et al. Implementation challenges and successes of a population-based colorectal cancer screening program: a qualitative study of stakeholder perspectives. Implement Sci, 2015, 10: 41.
24.	King DK, Shoup JA, Raebel MA, et al. Planning for implementation success using RE-AIM and CFIR frameworks: a qualitative study. Front Public Health, 2020, 8: 59.
25.	鲁春丽, 曹卉娟, 徐东, 等. 实施科学产生的背景、概念和国内外发展近况. 中国中西医结合杂志, 2020, 40(11): 1378-1380.
26.	Klesges LM, Estabrooks PA, Dzewaltowski DA, et al. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med, 2005, 29 Suppl: 66-75.
27.	Kessler RS, Purcell EP, Glasgow RE, et al. What does it mean to "employ" the RE-AIM model. Eval Health Prof, 2013, 36(1): 44-66.
28.	Galaviz KI, Harden SM, Smith E, et al. Physical activity promotion in Latin American populations: a systematic review on issues of internal and external validity. Int J Behav Nutr Phys Act, 2014, 11: 77.
29.	Hoehner CM, Ribeiro IC, Parra DC, et al. Physical activity interventions in Latin America: expanding and classifying the evidence. Am J Prev Med, 2013, 44(3): e31-e40.
30.	Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis, 2018, 15: E02.
31.	王拥军, 李子孝, 谷鸿秋, 等. 中国卒中报告2019(中文版)(1). 中国卒中杂志, 2020, 15(10): 1037-1043.
32.	何莎, 张田, 孙伟, 等. 中国内地脑卒中治疗多中心随机对照试验方法学质量现状报告. 华西医学, 2020, 35(6): 673-678.
33.	Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet, 1993, 342(8883): 1317-1322.
34.	Pronovost PJ. Enhancing physicians' use of clinical guidelines. JAMA, 2013, 310(23): 2501-2502.
35.	Liu M, Zhang C, Zha Q, et al. A national survey of Chinese medicine doctors and clinical practice guidelines in China. BMC Complement Altern Med, 2017, 17(1): 451.
36.	Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines. A framework for improvement. JAMA, 1999, 282(15): 1458-1465.
37.	Working Group Members for Stroke. International consortium for health outcomes measurement (ICHOM). Stroke data collection reference guide. 2018.
38.	RANKIN J. Cerebral vascular accidents in patients over the age of 60. Ⅱ. Prognosis. Scott Med J, 1957, 2(5): 200-215.
39.	Quinn TJ, Langhorne P, Stott DJ. Barthel index for stroke trials: development, properties, and application. Stroke, 2011, 42(4): 1146-1151.
40.	Goldstein LB, Bertels C, Davis JN. Interrater reliability of the NIH stroke scale. Arch Neurol, 1989, 46(6): 660-662.
41.	Vernon W. The Delphi technique: a review. Int J Ther Rehabil, 2013, 16(2): 69-76.
42.	Mullen R, Kydd A, Fleming A, et al. A practical guide to the systematic application of nominal group technique. Nurse Res, 2021, 29(1): 14-20.
43.	柯惠新, 黄京华, 沈浩. 调查研究中的统计分析法. 北京: 北京广播学院出版社, 1996: 351.
44.	黄敬亨. 健康教育学. 上海: 上海医科大学出版社, 1997: 212.
45.	柯惠新, 沈浩. 调查研究中的统计分析方法. 北京: 北京传媒大学出版社, 2005: 324.
46.	席仲恩, 杜珏 (译). 量表编制: 理论与应用(第三版). 重庆: 重庆大学出版社, 2016.
47.	Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decision Sci, 1997, 28(3): 763-774.
48.	杨宗义. 教育统计学(第1版). 北京: 科学技术文献出版社, 1990.
49.	鲁曼, 刘虹, 郑志杰. 医学生全球胜任力指标体系构建的研究. 中华医学教育杂志, 2021, 41(12): 1057-1061.
50.	Shen L, Yang J, Jin X, et al. Based on Delphi method and analytic hierarchy process to construct the evaluation index system of nursing simulation teaching quality. Nurse Educ Today, 2019, 79: 67-73.
51.	张天芳, 陈春芳. 关于Kendall协和系数的理解. 统计学与应用, 2020, 9(4): 578-581.
52.	Reilly KL, Kennedy S, Porter G, et al. Comparing, contrasting, and integrating dissemination and implementation outcomes included in the RE-AIM and implementation outcomes frameworks. Front Public Health, 2020, 8: 430.

1. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med, 2011, 104(12): 510-520.
2. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform, 2000, 9(1): 65-70.
3. 陈文嘉, 徐东, 李慧, 等. 实施科学理论的分类与介绍. 中国循证医学杂志, 2020, 20(8): 986-992.
4. 屈智勇, 郭帅, 张维军, 等. 实施科学对我国心理健康服务体系建设的启示. 北京师范大学学报(社会科学版), 2017, (2): 29-36.
5. Nutley S, Walter I, Davies HTO. From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation, 2003, 9(2): 125-148.
6. Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatry Res, 2020, 283: 112461.
7. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci, 2015, 10: 53.
8. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health, 1999, 89(9): 1322-1327.
9. Proctor EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health, 2009, 36(1): 24-34.
10. Green LW, Kreuter MW. Health promotion planning: an educational and ecological approach (3rd edition). Mountain View, CA: Mayfield Publishing, 1999.
11. Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2019, 7: 64.
12. Tabak RG, Khoong EC, Chambers DA, et al. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med, 2012, 43(3): 337-350.
13. Stange KC. Commentary: RE-AIM Planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2020, 8: 245.
14. King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health, 2010, 100(11): 2076-2084.
15. 中华中医药学会脑病分会, 广东省中医药学会脑病专业委员会, 广东省中西医结合学会卒中专业委员会. 中西医结合脑卒中循证实践指南(2019). 中国循证医学杂志, 2020, 20(8): 901-912.
16. Glasgow RE, Estabrooks PA, Ory MG. Characterizing evolving frameworks: issues from Esmail et al. (2020) review. Implement Sci, 2020, 15(1): 53.
17. Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol, 2004, 23(5): 443-451.
18. 娄晓民. RE-AIM模型. 余金明, 姜庆五, 主编. 现代健康教育学. 上海: 复旦大学出版社, 2019: 120-123.
19. Holtrop JS, Estabrooks PA, Gaglio B, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci, 2021, 5(1): e126.
20. 田文华. 基于RE-AIM模型的城乡居民大病保险模式评估. 复旦学报(社会科学版), 2021, 63(1): 152-160.
21. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health, 2013, 103(6): e38-e46.
22. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf, 2008, 34(4): 228-243.
23. Liles EG, Schneider JL, Feldstein AC, et al. Implementation challenges and successes of a population-based colorectal cancer screening program: a qualitative study of stakeholder perspectives. Implement Sci, 2015, 10: 41.
24. King DK, Shoup JA, Raebel MA, et al. Planning for implementation success using RE-AIM and CFIR frameworks: a qualitative study. Front Public Health, 2020, 8: 59.
25. 鲁春丽, 曹卉娟, 徐东, 等. 实施科学产生的背景、概念和国内外发展近况. 中国中西医结合杂志, 2020, 40(11): 1378-1380.
26. Klesges LM, Estabrooks PA, Dzewaltowski DA, et al. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med, 2005, 29 Suppl: 66-75.
27. Kessler RS, Purcell EP, Glasgow RE, et al. What does it mean to "employ" the RE-AIM model. Eval Health Prof, 2013, 36(1): 44-66.
28. Galaviz KI, Harden SM, Smith E, et al. Physical activity promotion in Latin American populations: a systematic review on issues of internal and external validity. Int J Behav Nutr Phys Act, 2014, 11: 77.
29. Hoehner CM, Ribeiro IC, Parra DC, et al. Physical activity interventions in Latin America: expanding and classifying the evidence. Am J Prev Med, 2013, 44(3): e31-e40.
30. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis, 2018, 15: E02.
31. 王拥军, 李子孝, 谷鸿秋, 等. 中国卒中报告2019(中文版)(1). 中国卒中杂志, 2020, 15(10): 1037-1043.
32. 何莎, 张田, 孙伟, 等. 中国内地脑卒中治疗多中心随机对照试验方法学质量现状报告. 华西医学, 2020, 35(6): 673-678.
33. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet, 1993, 342(8883): 1317-1322.
34. Pronovost PJ. Enhancing physicians' use of clinical guidelines. JAMA, 2013, 310(23): 2501-2502.
35. Liu M, Zhang C, Zha Q, et al. A national survey of Chinese medicine doctors and clinical practice guidelines in China. BMC Complement Altern Med, 2017, 17(1): 451.
36. Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines. A framework for improvement. JAMA, 1999, 282(15): 1458-1465.
37. Working Group Members for Stroke. International consortium for health outcomes measurement (ICHOM). Stroke data collection reference guide. 2018.
38. RANKIN J. Cerebral vascular accidents in patients over the age of 60. Ⅱ. Prognosis. Scott Med J, 1957, 2(5): 200-215.
39. Quinn TJ, Langhorne P, Stott DJ. Barthel index for stroke trials: development, properties, and application. Stroke, 2011, 42(4): 1146-1151.
40. Goldstein LB, Bertels C, Davis JN. Interrater reliability of the NIH stroke scale. Arch Neurol, 1989, 46(6): 660-662.
41. Vernon W. The Delphi technique: a review. Int J Ther Rehabil, 2013, 16(2): 69-76.
42. Mullen R, Kydd A, Fleming A, et al. A practical guide to the systematic application of nominal group technique. Nurse Res, 2021, 29(1): 14-20.
43. 柯惠新, 黄京华, 沈浩. 调查研究中的统计分析法. 北京: 北京广播学院出版社, 1996: 351.
44. 黄敬亨. 健康教育学. 上海: 上海医科大学出版社, 1997: 212.
45. 柯惠新, 沈浩. 调查研究中的统计分析方法. 北京: 北京传媒大学出版社, 2005: 324.
46. 席仲恩, 杜珏 (译). 量表编制: 理论与应用(第三版). 重庆: 重庆大学出版社, 2016.
47. Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decision Sci, 1997, 28(3): 763-774.
48. 杨宗义. 教育统计学(第1版). 北京: 科学技术文献出版社, 1990.
49. 鲁曼, 刘虹, 郑志杰. 医学生全球胜任力指标体系构建的研究. 中华医学教育杂志, 2021, 41(12): 1057-1061.
50. Shen L, Yang J, Jin X, et al. Based on Delphi method and analytic hierarchy process to construct the evaluation index system of nursing simulation teaching quality. Nurse Educ Today, 2019, 79: 67-73.
51. 张天芳, 陈春芳. 关于Kendall协和系数的理解. 统计学与应用, 2020, 9(4): 578-581.
52. Reilly KL, Kennedy S, Porter G, et al. Comparing, contrasting, and integrating dissemination and implementation outcomes included in the RE-AIM and implementation outcomes frameworks. Front Public Health, 2020, 8: 430.

Chinese Journal of Evidence-Based Medicine

Method of using RE-AIM framework to develop implementation outcome indicators

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content