Sleep stage classification is a necessary fundamental method for the diagnosis of sleep diseases, which has attracted extensive attention in recent years. Traditional methods for sleep stage classification, such as manual marking methods and machine learning algorithms, have the limitations of low efficiency and defective generalization. Recently, deep neural networks have shown improved results by the capability of learning complex pattern in the sleep data. However, these models ignore the intra-temporal sequential information and the correlation among all channels in each segment of the sleep data. To solve these problems, a hybrid attention temporal sequential network model is proposed in this paper, choosing recurrent neural network to replace traditional convolutional neural network, and extracting temporal features of polysomnography from the perspective of time. Furthermore, intra-temporal attention mechanism and channel attention mechanism are adopted to achieve the fusion of the intra-temporal representation and the fusion of channel-correlated representation. And then, based on recurrent neural network and inter-temporal attention mechanism, this model further realized the fusion of inter-temporal contextual representation. Finally, the end-to-end automatic sleep stage classification is accomplished according to the above hybrid representation. This paper evaluates the proposed model based on two public benchmark sleep datasets downloaded from open-source website, which include a number of polysomnography. Experimental results show that the proposed model could achieve better performance compared with ten state-of-the-art baselines. The overall accuracy of sleep stage classification could reach 0.801, 0.801 and 0.717, respectively. Meanwhile, the macro average F1-scores of the proposed model could reach 0.752, 0.728 and 0.700. All experimental results could demonstrate the effectiveness of the proposed model.