Cardiovascular disease (CVD) is one of the leading causes of death worldwide. Heart sound classification plays a key role in the early detection of CVD. The difference between normal and abnormal heart sounds is not obvious. In this paper, in order to improve the accuracy of the heart sound classification model, we propose a heart sound feature extraction method based on bispectral analysis and combine it with convolutional neural network (CNN) to classify heart sounds. The model can effectively suppress Gaussian noise by using bispectral analysis and can effectively extract the features of heart sound signals without relying on the accurate segmentation of heart sound signals. At the same time, the model combines with the strong classification performance of convolutional neural network and finally achieves the accurate classification of heart sound. According to the experimental results, the proposed algorithm achieves 0.910, 0.884 and 0.940 in terms of accuracy, sensitivity and specificity under the same data and experimental conditions, respectively. Compared with other heart sound classification algorithms, the proposed algorithm shows a significant improvement and strong robustness and generalization ability, so it is expected to be applied to the auxiliary detection of congenital heart disease.