Auscultation of heart sounds is an important method for the diagnosis of heart conditions. For most people, the audible component of heart sound are the first heart sound (S1) and the second heart sound (S2). Different diseases usually generate murmurs at different stages in a cardiac cycle. Segmenting the heart sounds precisely is the prerequisite for diagnosis. S1 and S2 emerges at the beginning of systole and diastole, respectively. Locating S1 and S2 accurately is beneficial for the segmentation of heart sounds. This paper proposed a method to classify the S1 and S2 based on their properties, and did not take use of the duration of systole and diastole. S1 and S2 in the training dataset were transformed to spectra by short-time Fourier transform and be feed to the two-stream convolutional neural network. The classification accuracy of the test dataset was as high as 91.135%. The highest sensitivity and specificity were 91.156% and 92.074%, respectively. Extracting the features of the input signals artificially can be avoid with the method proposed in this article. The calculation is not complicated, which makes this method effective for distinguishing S1 and S2 in real time.