BERT-based automated risk of bias assessment_Chinese Journal of Evidence-Based Medicine

Authors：

XIA Yuan , LIU Dongfeng , ZHANG Jinkui ,  LI Ke

School of Life Science & Technology, University of Electronic Science & Technology of China, Chengdu 610051, P.R.China;

Corresponding author：

LI Ke, Email: colinlike@163.com

Keywords：

Evidence-based medicine; Systematic review; Automated; Risk bias assessment; BERT

DOI：

10.7507/1672-2531.202006177

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To realize automatic risk bias assessment for the randomized controlled trial (RCT) literature using BERT (Bidirectional Encoder Representations from Transformers) as an approach for feature representation and text classification.Methods We first searched The Cochrane Library to obtain risk bias assessment data and detailed information on RCTs, and constructed data sets for text classification. We assigned 80% of the data set as the training set, 10% as the test set, and 10% as the validation set. Then, we used BERT to extract features, construct text classification model, and evaluate the seven types of risk bias values (high and low). The results were compared with those from traditional machine learning methods using a combination of n-gram and TF-IDF as well as the Linear SVM classifier. The accuracy rate (P value), recall rate (R value) and F₁ value were used to evaluate the performance of the models.Results Our BERT-based model achieved F₁ values of 78.5% to 95.2% for the seven types of risk bias assessment tasks, which was 14.7% higher than the traditional machine learning method. F₁ values of 85.7% to 92.8% were obtained in the extraction task of the other six types of biased descriptors except "other sources of bias", which was 18.2% higher than the traditional machine learning method.Conclusions The BERT-based automatic risk bias assessment model can realize higher accuracy in risk of bias assessment for RCT literature, and improve the efficiency of assessment.

Citation： XIA Yuan, LIU Dongfeng, ZHANG Jinkui, LI Ke. BERT-based automated risk of bias assessment. Chinese Journal of Evidence-Based Medicine, 2021, 21(2): 204-209. doi: 10.7507/1672-2531.202006177 Copy

1.	周巧珍, 周昱, 田柳, 等. 氧化锆种植体基台机械性能及存活率的系统评价. 中国口腔种植学杂志, 2014, (3): 116-121.
2.	Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ, 2011, 343: d5928.
3.	Savović J, Weeks L, Sterne J AC, et al. Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev, 2014, 3(1): 37.
4.	Hartling L, Hamm MP, Milne A, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol, 2013, 66(9): 973-981.
5.	Armijo-Olivo S, Ospina M, da Costa BR, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One, 2014, 9(5): e96920.
6.	Tsafnat G, Glasziou P, Choong MK, et al. Systematic review automation technologies. Syst Rev, 2014, 3(1): 74.
7.	Hassanzadeh H, Groza T, Hunter J. Identifying scientific artefacts in biomedical literature: the evidence based medicine use case. J Biomed Inform, 2014, 49: 159-170.
8.	Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. Los Angeles: Acm Conference on Bioinformatics, 2014: 88-95.
9.	Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trialsJ. IEEE J Biomed Health Inform, 2015, 19(4): 1406-1412.
10.	Millard LA, Flach PA, Higgins JP. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol, 2016, 45(1): 266-277.
11.	Mintz M, Bills S, Snow R, et al. Distant supervision for relation extraction without labeled data. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2. Association for Computational Linguistics, 2009: 1003-1011.
12.	Nguyen TVT, Moschitti A. End-to-end relation extraction using distant supervision from external semantic repositories. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, 2011: 277-282.
13.	Cochrane Collaboration. The Cochrane database of systematic reviews, 2014. Available at: http://www.thecochranelibrary.com.
14.	Wang J, Zhou Q, Sun G. Text Similarity Based on Semantic Analysis. 2016 2nd International Conference on Artificial Intelligence and Industrial Engineering (AIIE 2016). Atlanta: Atlantis Press, 2016: 303-307.
15.	Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. Available at: https://arxiv.org/abs/1810.04805.

1. 周巧珍, 周昱, 田柳, 等. 氧化锆种植体基台机械性能及存活率的系统评价. 中国口腔种植学杂志, 2014, (3): 116-121.
2. Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ, 2011, 343: d5928.
3. Savović J, Weeks L, Sterne J AC, et al. Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev, 2014, 3(1): 37.
4. Hartling L, Hamm MP, Milne A, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol, 2013, 66(9): 973-981.
5. Armijo-Olivo S, Ospina M, da Costa BR, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One, 2014, 9(5): e96920.
6. Tsafnat G, Glasziou P, Choong MK, et al. Systematic review automation technologies. Syst Rev, 2014, 3(1): 74.
7. Hassanzadeh H, Groza T, Hunter J. Identifying scientific artefacts in biomedical literature: the evidence based medicine use case. J Biomed Inform, 2014, 49: 159-170.
8. Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. Los Angeles: Acm Conference on Bioinformatics, 2014: 88-95.
9. Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trialsJ. IEEE J Biomed Health Inform, 2015, 19(4): 1406-1412.
10. Millard LA, Flach PA, Higgins JP. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol, 2016, 45(1): 266-277.
11. Mintz M, Bills S, Snow R, et al. Distant supervision for relation extraction without labeled data. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2. Association for Computational Linguistics, 2009: 1003-1011.
12. Nguyen TVT, Moschitti A. End-to-end relation extraction using distant supervision from external semantic repositories. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, 2011: 277-282.
13. Cochrane Collaboration. The Cochrane database of systematic reviews, 2014. Available at: http://www.thecochranelibrary.com.
14. Wang J, Zhou Q, Sun G. Text Similarity Based on Semantic Analysis. 2016 2nd International Conference on Artificial Intelligence and Industrial Engineering (AIIE 2016). Atlanta: Atlantis Press, 2016: 303-307.
15. Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. Available at: https://arxiv.org/abs/1810.04805.

Chinese Journal of Evidence-Based Medicine

BERT-based automated risk of bias assessment

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content