ObjectivesTo explore the effect of the deep learning algorithm convolutional neural network (CNN) in screening of randomized controlled trials (RCTs) in Chinese medical literatures.MethodsLiterature with the topic " oral science” published in 2014 were retrieved from CNKI and exported citations containing title and abstract. RCTs screening was conducted by double independent screening, checking and peer discussion. The final results of the citations were used for CNN algorithm model training. After completing the algorithm model training, a prospective comparative trial was organized by searching all literature with the topic "oral science" published in CNKI from January to March 2018 to compare the sensitivity (SEN) and specificity (SPE) of algorithm with manual screening. The initial results of a single screener represented the performance of manual screening, and the final results after peer discussion were used as the gold standard. The best thresholds of algorithm were determined with the receptor operative characteristic (ROC) curve.ResultsA total of 1 246 RCTs and 4 754 non-RCTs were eventually included for training and testing of CNN algorithm model. 249 RCTs and 949 non-RCTs were included in the prospective trial. The SEN and SPE of manual screening were 98.01% and 98.82%. For the algorithm model, the SEN of RCTs screening decreased with the increase of threshold value while the SPE increased with the increase of threshold value. After 27 changes of threshold value, ROC curve were obtained. The area under the ROC curve was 0.9977, unveiling the optimal accuracy threshold (Threshold=0.4, SEN=98.39%, SPE=98.84%) and high sensitivity threshold (Threshold=0.06, SEN=99.60%, SPE=94.10%).ConclusionsA CNN algorithm model is trained with Chinese RCTs classification database established in this study and shows an excellent classification performance in screening RCTs of Chinese medical literature, which is proved to be comparable to the manual screening performance in the prospective controlled trial.