It has been absent from an accepted criteria for normalization and quality control of the thoracic surgery until now. The ideal assessing instrument which will be used to evaluate the technical skills and surgical procedures should present a few vital characterizations below: objectivity, speciality in the content, detailed structure, and quantifiability. Objective structured assessment of technical skills (OSATS) has developed as a reliable method of surgical skills measurement. This article focuses on the history of OSATS and its prospect in the thoracic surgery area by reviewing relevant literatures.
Citation: LI Hui, YOU Bin, YE Xin. Research progress and prospect of objective structured assessment of technical skills (OSATS) in the quality control of thoracic surgical procedures. Chinese Journal of Clinical Thoracic and Cardiovascular Surgery, 2019, 26(1): 35-41. doi: 10.7507/1007-4848.201810040 Copy
1. | Reznick R, Regehr G, MacRae H, et al. Testing technical skill via an innovative "bench station" examination. Am J Surg, 1997, 173(3): 226-230. |
2. | Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 1997, 84(2): 273-278. |
3. | Datta V, Bann S, Beard J, et al. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg, 2004, 199(4): 603-606. |
4. | Goff B, Mandel L, Lentz G, et al. Assessment of resident surgical skills: is testing feasible? Am J Obstet Gynecol, 2005, 192(4): 1331-1338. |
5. | Datta V, Chang A, Mackay S, et al. The relationship between motion analysis and surgical technical assessments. Am J Surg, 2002, 184(1): 70-73. |
6. | Hernandez JD, Bann SD, Munz Y, et al. Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surg Endosc, 2004, 18(3): 372-378. |
7. | Johnston JL, Lundy G, McCullough M, et al. The view from over there: reframing the OSCE through the experience of standardised patient raters. Med Educ, 2013, 47(9): 899-909. |
8. | Bingener J, Boyd T, Van Sickle K, et al. Randomized double-blinded trial investigating the impact of a curriculum focused on error recognition on laparoscopic suturing training. Am J Surg, 2008, 195(2): 179-182. |
9. | Steigerwald SN, Park J, Hardy KM, et al. Establishing the concurrent validity of general and technique-specific skills assessments in surgical education. Am J Surg, 2016, 211(1): 268-273. |
10. | Mavis BE, Henry RC, Ogle KS, et al. The emperor's new clothes: the OSCE reassessed. Acad Med, 1996, 71(5): 447-453. |
11. | van Hove PD, Tuijthof GJ, Verdaasdonk EG, et al. Objective assessment of technical surgical skills. Br J Surg, 2010, 97(7): 972-987. |
12. | Dumestre D, Yeung JK, Temple-Oberle C. Evidence-based microsurgical skills acquisition series part 2: validated assessment instruments--a systematic review. J Surg Educ, 2015, 72(1): 80-89. |
13. | de Montbrun S, Roberts PL, Satterthwaite L, et al. Implementing and evaluating a national certification technical skills examination: The colorectal objective structured assessment of technical skill. Ann Surg, 2016, 264(1): 1-6. |
14. | Zevin B, Bonrath EM, Aggarwal R, et al. Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg, 2013, 216(5): 955-965. |
15. | Al-Jabir A, Aydin A, Abe T, et al. Validation of the advanced scope trainer for flexible ureterorenoscopy training. Urology, 2017, 110: 45-50. |
16. | Noureldin YA, Elkoushy MA, Aloosh M, et al. Objective structured assessment of technical skills for the photoselective vaporization of the prostate procedure: a pilot study. J Endourol, 2016, 30(8): 923-929. |
17. | Moktar J, Popkin CA, Howard A, et al. Development of a cast application simulator and evaluation of objective measures of performance. J Bone Joint Surg Am, 2014, 96(9): e76. |
18. | Wilasrusmee C, Lertsithichai P, Kittur DS. Vascular anastomosis model: relation between competency in a laboratory-based model and surgical competency. Eur J Vasc Endovasc Surg, 2007, 34(4): 405-410. |
19. | Bhatti NI. Assessment of surgical skills and competency. Otolaryngol Clin North Am, 2017, 50(5): 959-965. |
20. | Palter VN, Orzech N, Reznick RK, et al. Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg, 2013, 257(2): 224-230. |
21. | Aoun SG, El Ahmadieh TY, El Tecle NE, et al. A pilot study to assess the construct and face validity of the Northwestern Objective Microanastomosis Assessment Tool. J Neurosurg, 2015, 123(1): 103-109. |
22. | Chahine EB, Han CH, Auguste T. Construct validity of a simple laparoscopic ovarian cystectomy model using a validated objective structured assessment of technical skills. J Minim Invasive Gynecol, 2017, 24(5): 850-854. |
23. | Aldave G, Hansen D, Briceño V, et al. Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery. J Neurosurg Pediatr, 2017, 19(4): 377-383. |
24. | Kaijser M, van Ramshorst G, van Wagensveld B, et al. Current techniques of teaching and learning in bariatric surgical procedures: a systematic review. J Surg Educ, 2018, 75(3): 730-738. |
25. | Ghaderi I, Auvergne L, Park YS, et al. Quantitative and qualitative analysis of performance during advanced laparoscopic fellowship: a curriculum based on structured assessment and feedback. Am J Surg, 2015, 209(1): 71-78. |
26. | Lodge D, Grantcharov T. Training and assessment of technical skills and competency in cardiac surgery. Eur J Cardiothorac Surg, 2011, 39(3): 287-293. |
27. | Polin MR, Siddiqui NY, Comstock BA, et al. Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol, 2016, 215(5): 644. |
28. | Nielsen PE, Foglia LM, Mandel LS, et al. Objective structured assessment of technical skills for episiotomy repair. Am J Obstet Gynecol, 2003, 189(5): 1257-1260. |
29. | Ghaderi I, Manji F, Park YS, et al. Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg, 2015, 261(2): 251-262. |
30. | Quick JA, Kudav V, Doty J, et al. Surgical resident technical skill self-evaluation: increased precision with training progression. J Surg Res, 2017, 218: 144-149. |
31. | De La Garza JR, Schmidt MW, Kowalewski KF, et al. Does rating with a checklist improve the effect of E-learning for cognitive and practical skills in bariatric surgery? A rater-blinded, randomized-controlled trial. Surg Endosc, 2018, [Epub ahead of print]. |
32. | Bhatti NI, Ahmed A. Improving skills development in residency using a deliberate-practice and learner-centered model. Laryngoscope, 2015, 125 Suppl 8: S1-S14. |
33. | Bouma W, Kuijpers M, Bijleveld A, et al. A new beating-heart off-pump coronary artery bypass grafting training model. Interact Cardiovasc Thorac Surg, 2015, 20(1): 7-9. |
34. | Jensen AR, Wright AS, Levy AE, et al. Acquiring basic surgical skills: is a faculty mentor really needed? Am J Surg, 2009, 197(1): 82-88. |
35. | Soucisse ML, Boulva K, Sideris L, et al. Video coaching as an efficient teaching method for surgical residents-a randomized controlled trial. J Surg Educ, 2017, 74(2): 365-371. |
36. | Bayer J, Moulton CA, Monden K, et al. The role of multiorgan procurement for abdominal transplant in general surgery resident education. Am J Surg, 2018, 216(2): 331-336. |
37. | Karmali RJ, Siu JM, You DZ, et al. The Surgical Skills and Technology Elective Program (SSTEP): A comprehensive simulation-based surgical skills initiative for preclerkship medical students. Am J Surg, 2018, 216(2): 375-381. |
38. | Romero P, Günther P, Kowalewski KF, et al. Halsted's "See One, Do One, and Teach One" versus Peyton's Four-Step Approach: a randomized trial for training of laparoscopic suturing and knot tying. J Surg Educ, 2018, 75(2): 510-515. |
39. | Sharma G, Aycart MA, O'Mara L, et al. A cadaveric procedural anatomy simulation course improves video-based assessment of operative performance. J Surg Res, 2018, 223: 64-71. |
40. | Singh P, Aggarwal R, Tahir M, et al. A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills. Ann Surg, 2015, 261(5): 862-869. |
41. | Nesbitt JC, St Julien J, Absi TS, et al. Tissue-based coronary surgery simulation: medical student deliberate practice can achieve equivalency to senior surgery residents. J Thorac Cardiovasc Surg, 2013, 145(6): 1453-1458. |
42. | Sobel RH, Blanco R, Ha PK, et al. Implementation of a comprehensive competency-based transoral robotic surgery training curriculum with ex vivo dissection models. Head Neck, 2016, 38(10): 1553-1563. |
43. | Willems JI, Shin AM, Shin DM, et al. A comparison of robotically assisted microsurgery versus manual microsurgery in challenging situations. Plast Reconstr Surg, 2016, 137(4): 1317-1324. |
44. | Ballouhey Q, Clermidi P, Cros J, et al. Comparison of 8 and 5 mm robotic instruments in small cavities: 5 or 8 mm robotic instruments for small cavities? Surg Endosc, 2018, 32(2): 1027-1034. |
45. | Robison W, Patel SK, Mehta A, et al. Can fatigue affect acquisition of new surgical skills? A prospective trial of pre- and post-call general surgery residents using the da Vinci surgical skills simulator. Surg Endosc, 2018, 32(3): 1389-1396. |
46. | Ault G, Reznick R, MacRae H, et al. Exporting a technical skills evaluation technology to other sites. Am J Surg, 2001, 182(3): 254-256. |
47. | Nickel F, Hendrie JD, Stock C, et al. Direct observation versus endoscopic video recording-based rating with the objective structured assessment of technical skills for training of laparoscopic cholecystectomy. Eur Surg Res, 2016, 57(1-2): 1-9. |
48. | Hopmans CJ, den Hoed PT, van der Laan L, et al. Assessment of surgery residents' operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study. Surgery, 2014, 156(5): 1078-1088. |
49. | Hu WG, Feng JY, Wang J, et al. Ureteroscopy and cystoscopy training: comparison between transparent and non-transparent simulators. BMC Med Educ, 2015, 15: 93. |
50. | Steigerwald SN, Park J, Hardy KM, et al. Does laparoscopic simulation predict intraoperative performance? A comparison between the Fundamentals of Laparoscopic Surgery and LapVR evaluation metrics. Am J Surg, 2015, 209(1): 34-39. |
51. | Tam V, Zenati M, Novak S, et al. Robotic pancreatoduodenectomy biotissue curriculum has validity and improves technical performance for surgical oncology fellows. J Surg Educ, 2017, 74(6): 1057-1065. |
52. | Szasz P, Bonrath EM, Louridas M, et al. Setting performance standards for technical and nontechnical competence in general surgery. Ann Surg, 2017, 266(1): 1-7. |
53. | Wetzel CM, George A, Hanna GB, et al. Stress management training for surgeons--a randomized, controlled, intervention study. Ann Surg, 2011, 253(3): 488-494. |
54. | Fecso AB, Kuzulugil SS, Babaoglu C, et al. Relationship between intraoperative non-technical performance and technical events in bariatric surgery. Br J Surg, 2018, 105(8): 1044-1050. |
55. | Kwong JC, Lee JY, Goldenberg MG. Understanding and assessing nontechnical skills in robotic urological surgery: a systematic review and synthesis of the validity evidence. J Surg Educ, 2018, [Epub ahead of print]. |
56. | Wongworawat MD. Editor's Spotlight/Take 5: Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively. Clin Orthop Relat Res, 2016, 474(4): 871-873. |
57. | Hogg ME, Zenati M, Novak S, et al. Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy tndependent of patient-related variables. Ann Surg, 2016, 264(3): 482-491. |
58. | Fecso AB, Bhatti JA, Stotland PK, et al. Technical performance as a predictor of clinical outcomes in laparoscopic gastric cancer surgery. Ann Surg, 2018, [Epub ahead of print]. |
59. | Varban OA, Greenberg CC, Schram J, et al. Surgical skill in bariatric surgery: Does skill in one procedure predict outcomes for another? Surgery, 2016, 160(5): 1172-1181. |
60. | Azari DP, Frasier LL, Quamme SRP, et al. Modeling surgical technical skill using expert assessment for automated computer rating. Ann Surg, 2017, [Epub ahead of print]. |
61. | Strøm M, Lönn L, Bech B, et al. Assessment of competence in EVAR procedures: a novel rating scale developed by the Delphi technique. Eur J Vasc Endovasc Surg, 2017, 54(1): 34-41. |
62. | Oquendo YA, Riddle EW, Hiller D, et al. Automatically rating trainee skill at a pediatric laparoscopic suturing task. Surg Endosc, 2018, 32(4): 1840-1857. |
63. | Kumar R, Jog A, Vagvolgyi B, et al. Objective measures for longitudinal assessment of robotic surgery training. J Thorac Cardiovasc Surg, 2012, 143(3): 528-534. |
64. | Kowalewski KF, Hendrie JD, Schmidt MW, et al. Development and validation of a sensor- and expert model-based training system for laparoscopic surgery: the iSurgeon. Surg Endosc, 2017, 31(5): 2155-2165. |
65. | Barsness KA, Rooney DM, Davis LM, et al. Validation of measures from a thoracoscopic esophageal atresia/tracheoesophageal fistula repair simulator. J Pediatr Surg, 2014, 49(1): 29-32. |
66. | Friedrich M, Bergdolt C, Haubruck P, et al. App-based serious gaming for training of chest tube insertion: study protocol for a randomized controlled trial. Trials, 2017, 18(1): 56. |
67. | Bohnen JD, Demetri L, Fuentes E, et al. High-fidelity emergency department thoracotomy simulator with beating-heart technology and OSATS tool improves trainee confidence and distinguishes level of skill. J Surg Educ, 2018, 75(5): 1357-1366. |
68. | Kenny L, Booth K, Freystaetter K, et al. Training cardiothoracic surgeons of the future: The UK experience. J Thorac Cardiovasc Surg, 2018, 155(6): 2526-2538. |
69. | Jensen K, Petersen RH, Hansen HJ, et al. A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy. Surg Endosc, 2018, 32(10): 4173-4182. |
- 1. Reznick R, Regehr G, MacRae H, et al. Testing technical skill via an innovative "bench station" examination. Am J Surg, 1997, 173(3): 226-230.
- 2. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 1997, 84(2): 273-278.
- 3. Datta V, Bann S, Beard J, et al. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg, 2004, 199(4): 603-606.
- 4. Goff B, Mandel L, Lentz G, et al. Assessment of resident surgical skills: is testing feasible? Am J Obstet Gynecol, 2005, 192(4): 1331-1338.
- 5. Datta V, Chang A, Mackay S, et al. The relationship between motion analysis and surgical technical assessments. Am J Surg, 2002, 184(1): 70-73.
- 6. Hernandez JD, Bann SD, Munz Y, et al. Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surg Endosc, 2004, 18(3): 372-378.
- 7. Johnston JL, Lundy G, McCullough M, et al. The view from over there: reframing the OSCE through the experience of standardised patient raters. Med Educ, 2013, 47(9): 899-909.
- 8. Bingener J, Boyd T, Van Sickle K, et al. Randomized double-blinded trial investigating the impact of a curriculum focused on error recognition on laparoscopic suturing training. Am J Surg, 2008, 195(2): 179-182.
- 9. Steigerwald SN, Park J, Hardy KM, et al. Establishing the concurrent validity of general and technique-specific skills assessments in surgical education. Am J Surg, 2016, 211(1): 268-273.
- 10. Mavis BE, Henry RC, Ogle KS, et al. The emperor's new clothes: the OSCE reassessed. Acad Med, 1996, 71(5): 447-453.
- 11. van Hove PD, Tuijthof GJ, Verdaasdonk EG, et al. Objective assessment of technical surgical skills. Br J Surg, 2010, 97(7): 972-987.
- 12. Dumestre D, Yeung JK, Temple-Oberle C. Evidence-based microsurgical skills acquisition series part 2: validated assessment instruments--a systematic review. J Surg Educ, 2015, 72(1): 80-89.
- 13. de Montbrun S, Roberts PL, Satterthwaite L, et al. Implementing and evaluating a national certification technical skills examination: The colorectal objective structured assessment of technical skill. Ann Surg, 2016, 264(1): 1-6.
- 14. Zevin B, Bonrath EM, Aggarwal R, et al. Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg, 2013, 216(5): 955-965.
- 15. Al-Jabir A, Aydin A, Abe T, et al. Validation of the advanced scope trainer for flexible ureterorenoscopy training. Urology, 2017, 110: 45-50.
- 16. Noureldin YA, Elkoushy MA, Aloosh M, et al. Objective structured assessment of technical skills for the photoselective vaporization of the prostate procedure: a pilot study. J Endourol, 2016, 30(8): 923-929.
- 17. Moktar J, Popkin CA, Howard A, et al. Development of a cast application simulator and evaluation of objective measures of performance. J Bone Joint Surg Am, 2014, 96(9): e76.
- 18. Wilasrusmee C, Lertsithichai P, Kittur DS. Vascular anastomosis model: relation between competency in a laboratory-based model and surgical competency. Eur J Vasc Endovasc Surg, 2007, 34(4): 405-410.
- 19. Bhatti NI. Assessment of surgical skills and competency. Otolaryngol Clin North Am, 2017, 50(5): 959-965.
- 20. Palter VN, Orzech N, Reznick RK, et al. Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg, 2013, 257(2): 224-230.
- 21. Aoun SG, El Ahmadieh TY, El Tecle NE, et al. A pilot study to assess the construct and face validity of the Northwestern Objective Microanastomosis Assessment Tool. J Neurosurg, 2015, 123(1): 103-109.
- 22. Chahine EB, Han CH, Auguste T. Construct validity of a simple laparoscopic ovarian cystectomy model using a validated objective structured assessment of technical skills. J Minim Invasive Gynecol, 2017, 24(5): 850-854.
- 23. Aldave G, Hansen D, Briceño V, et al. Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery. J Neurosurg Pediatr, 2017, 19(4): 377-383.
- 24. Kaijser M, van Ramshorst G, van Wagensveld B, et al. Current techniques of teaching and learning in bariatric surgical procedures: a systematic review. J Surg Educ, 2018, 75(3): 730-738.
- 25. Ghaderi I, Auvergne L, Park YS, et al. Quantitative and qualitative analysis of performance during advanced laparoscopic fellowship: a curriculum based on structured assessment and feedback. Am J Surg, 2015, 209(1): 71-78.
- 26. Lodge D, Grantcharov T. Training and assessment of technical skills and competency in cardiac surgery. Eur J Cardiothorac Surg, 2011, 39(3): 287-293.
- 27. Polin MR, Siddiqui NY, Comstock BA, et al. Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol, 2016, 215(5): 644.
- 28. Nielsen PE, Foglia LM, Mandel LS, et al. Objective structured assessment of technical skills for episiotomy repair. Am J Obstet Gynecol, 2003, 189(5): 1257-1260.
- 29. Ghaderi I, Manji F, Park YS, et al. Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg, 2015, 261(2): 251-262.
- 30. Quick JA, Kudav V, Doty J, et al. Surgical resident technical skill self-evaluation: increased precision with training progression. J Surg Res, 2017, 218: 144-149.
- 31. De La Garza JR, Schmidt MW, Kowalewski KF, et al. Does rating with a checklist improve the effect of E-learning for cognitive and practical skills in bariatric surgery? A rater-blinded, randomized-controlled trial. Surg Endosc, 2018, [Epub ahead of print].
- 32. Bhatti NI, Ahmed A. Improving skills development in residency using a deliberate-practice and learner-centered model. Laryngoscope, 2015, 125 Suppl 8: S1-S14.
- 33. Bouma W, Kuijpers M, Bijleveld A, et al. A new beating-heart off-pump coronary artery bypass grafting training model. Interact Cardiovasc Thorac Surg, 2015, 20(1): 7-9.
- 34. Jensen AR, Wright AS, Levy AE, et al. Acquiring basic surgical skills: is a faculty mentor really needed? Am J Surg, 2009, 197(1): 82-88.
- 35. Soucisse ML, Boulva K, Sideris L, et al. Video coaching as an efficient teaching method for surgical residents-a randomized controlled trial. J Surg Educ, 2017, 74(2): 365-371.
- 36. Bayer J, Moulton CA, Monden K, et al. The role of multiorgan procurement for abdominal transplant in general surgery resident education. Am J Surg, 2018, 216(2): 331-336.
- 37. Karmali RJ, Siu JM, You DZ, et al. The Surgical Skills and Technology Elective Program (SSTEP): A comprehensive simulation-based surgical skills initiative for preclerkship medical students. Am J Surg, 2018, 216(2): 375-381.
- 38. Romero P, Günther P, Kowalewski KF, et al. Halsted's "See One, Do One, and Teach One" versus Peyton's Four-Step Approach: a randomized trial for training of laparoscopic suturing and knot tying. J Surg Educ, 2018, 75(2): 510-515.
- 39. Sharma G, Aycart MA, O'Mara L, et al. A cadaveric procedural anatomy simulation course improves video-based assessment of operative performance. J Surg Res, 2018, 223: 64-71.
- 40. Singh P, Aggarwal R, Tahir M, et al. A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills. Ann Surg, 2015, 261(5): 862-869.
- 41. Nesbitt JC, St Julien J, Absi TS, et al. Tissue-based coronary surgery simulation: medical student deliberate practice can achieve equivalency to senior surgery residents. J Thorac Cardiovasc Surg, 2013, 145(6): 1453-1458.
- 42. Sobel RH, Blanco R, Ha PK, et al. Implementation of a comprehensive competency-based transoral robotic surgery training curriculum with ex vivo dissection models. Head Neck, 2016, 38(10): 1553-1563.
- 43. Willems JI, Shin AM, Shin DM, et al. A comparison of robotically assisted microsurgery versus manual microsurgery in challenging situations. Plast Reconstr Surg, 2016, 137(4): 1317-1324.
- 44. Ballouhey Q, Clermidi P, Cros J, et al. Comparison of 8 and 5 mm robotic instruments in small cavities: 5 or 8 mm robotic instruments for small cavities? Surg Endosc, 2018, 32(2): 1027-1034.
- 45. Robison W, Patel SK, Mehta A, et al. Can fatigue affect acquisition of new surgical skills? A prospective trial of pre- and post-call general surgery residents using the da Vinci surgical skills simulator. Surg Endosc, 2018, 32(3): 1389-1396.
- 46. Ault G, Reznick R, MacRae H, et al. Exporting a technical skills evaluation technology to other sites. Am J Surg, 2001, 182(3): 254-256.
- 47. Nickel F, Hendrie JD, Stock C, et al. Direct observation versus endoscopic video recording-based rating with the objective structured assessment of technical skills for training of laparoscopic cholecystectomy. Eur Surg Res, 2016, 57(1-2): 1-9.
- 48. Hopmans CJ, den Hoed PT, van der Laan L, et al. Assessment of surgery residents' operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study. Surgery, 2014, 156(5): 1078-1088.
- 49. Hu WG, Feng JY, Wang J, et al. Ureteroscopy and cystoscopy training: comparison between transparent and non-transparent simulators. BMC Med Educ, 2015, 15: 93.
- 50. Steigerwald SN, Park J, Hardy KM, et al. Does laparoscopic simulation predict intraoperative performance? A comparison between the Fundamentals of Laparoscopic Surgery and LapVR evaluation metrics. Am J Surg, 2015, 209(1): 34-39.
- 51. Tam V, Zenati M, Novak S, et al. Robotic pancreatoduodenectomy biotissue curriculum has validity and improves technical performance for surgical oncology fellows. J Surg Educ, 2017, 74(6): 1057-1065.
- 52. Szasz P, Bonrath EM, Louridas M, et al. Setting performance standards for technical and nontechnical competence in general surgery. Ann Surg, 2017, 266(1): 1-7.
- 53. Wetzel CM, George A, Hanna GB, et al. Stress management training for surgeons--a randomized, controlled, intervention study. Ann Surg, 2011, 253(3): 488-494.
- 54. Fecso AB, Kuzulugil SS, Babaoglu C, et al. Relationship between intraoperative non-technical performance and technical events in bariatric surgery. Br J Surg, 2018, 105(8): 1044-1050.
- 55. Kwong JC, Lee JY, Goldenberg MG. Understanding and assessing nontechnical skills in robotic urological surgery: a systematic review and synthesis of the validity evidence. J Surg Educ, 2018, [Epub ahead of print].
- 56. Wongworawat MD. Editor's Spotlight/Take 5: Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively. Clin Orthop Relat Res, 2016, 474(4): 871-873.
- 57. Hogg ME, Zenati M, Novak S, et al. Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy tndependent of patient-related variables. Ann Surg, 2016, 264(3): 482-491.
- 58. Fecso AB, Bhatti JA, Stotland PK, et al. Technical performance as a predictor of clinical outcomes in laparoscopic gastric cancer surgery. Ann Surg, 2018, [Epub ahead of print].
- 59. Varban OA, Greenberg CC, Schram J, et al. Surgical skill in bariatric surgery: Does skill in one procedure predict outcomes for another? Surgery, 2016, 160(5): 1172-1181.
- 60. Azari DP, Frasier LL, Quamme SRP, et al. Modeling surgical technical skill using expert assessment for automated computer rating. Ann Surg, 2017, [Epub ahead of print].
- 61. Strøm M, Lönn L, Bech B, et al. Assessment of competence in EVAR procedures: a novel rating scale developed by the Delphi technique. Eur J Vasc Endovasc Surg, 2017, 54(1): 34-41.
- 62. Oquendo YA, Riddle EW, Hiller D, et al. Automatically rating trainee skill at a pediatric laparoscopic suturing task. Surg Endosc, 2018, 32(4): 1840-1857.
- 63. Kumar R, Jog A, Vagvolgyi B, et al. Objective measures for longitudinal assessment of robotic surgery training. J Thorac Cardiovasc Surg, 2012, 143(3): 528-534.
- 64. Kowalewski KF, Hendrie JD, Schmidt MW, et al. Development and validation of a sensor- and expert model-based training system for laparoscopic surgery: the iSurgeon. Surg Endosc, 2017, 31(5): 2155-2165.
- 65. Barsness KA, Rooney DM, Davis LM, et al. Validation of measures from a thoracoscopic esophageal atresia/tracheoesophageal fistula repair simulator. J Pediatr Surg, 2014, 49(1): 29-32.
- 66. Friedrich M, Bergdolt C, Haubruck P, et al. App-based serious gaming for training of chest tube insertion: study protocol for a randomized controlled trial. Trials, 2017, 18(1): 56.
- 67. Bohnen JD, Demetri L, Fuentes E, et al. High-fidelity emergency department thoracotomy simulator with beating-heart technology and OSATS tool improves trainee confidence and distinguishes level of skill. J Surg Educ, 2018, 75(5): 1357-1366.
- 68. Kenny L, Booth K, Freystaetter K, et al. Training cardiothoracic surgeons of the future: The UK experience. J Thorac Cardiovasc Surg, 2018, 155(6): 2526-2538.
- 69. Jensen K, Petersen RH, Hansen HJ, et al. A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy. Surg Endosc, 2018, 32(10): 4173-4182.