Automatic classification of heart sounds plays an important role in the early diagnosis of congenital heart disease. A kind of heart sound classification algorithms based on sub-band envelope feature and convolution neural network was proposed in this paper, which did not need to segment the heart sounds according to cardiac cycle accurately. Firstly, the heart sound signal was divided into some frames. Then, the frame level heart sound signal was filtered with Gammatone filter bank to obtain the sub-band signals. Next, the sub-band envelope was extracted by Hilbert transform. After that, the sub-band envelope was stacked into a feature map. Finally, type Ⅰ and type Ⅱ convolution neural network were selected as classifier. The result shown that the sub-band envelope feature was better in type Ⅰ than type Ⅱ. The algorithm is tested with 1 000 heart sound samples. The test results show that the overall performance of the algorithm proposed in this paper is significantly improved compared with other similar algorithms, which provides a new method for automatic classification of congenital heart disease, and speeds up the process of automatic classification of heart sounds applied to the actual screening.