Recently, deep neural networks (DNNs) have been widely used in the field of electrocardiogram (ECG) signal classification, but the previous models have limited ability to extract features from raw ECG data. In this paper, a deep residual network model based on pyramidal convolutional layers (PC-DRN) was proposed to implement ECG signal classification. The pyramidal convolutional (PC) layer could simultaneously extract multi-scale features from the original ECG data. And then, a deep residual network was designed to train the classification model for arrhythmia detection. The public dataset provided by the physionet computing in cardiology challenge 2017(CinC2017) was used to validate the classification experiment of 4 types of ECG data. In this paper, the harmonic mean F1 of classification accuracy and recall was selected as the evaluation indexes. The experimental results showed that the average sequence level F1 (SeqF1) of PC-DRN was improved from 0.857 to 0.920, and the average set level F1 (SetF1) was improved from 0.876 to 0.925. Therefore, the PC-DRN model proposed in this paper provided a promising way for the feature extraction and classification of ECG signals, and provided an effective tool for arrhythmia classification.