In the current LSTM-based video summary generation methods, memory units in the LSTM network cannot focus on the span of long time series when the input sequence is too long. A video summary generation model based on Recurrent Long short-term memory network (ReLSTM) and sequential attention (SSA) is proposed to improve the ability of deep learning networks to learn temporal features. This model uses the ReLSTM network to extract temporal features. Meanwhile, SSA is used to dynamically adjust the feature weights of each video sequence input into the ReLSTM network. The experimental results show that F1-score is increased by 2.5% on average and 0.2% at the highest on the dataset TVSum. On the dataset SumMe, F1-score was improved by 7.8% on average and 3.4% at the highest. Experimental results show that this method can effectively learn the temporal features between shots.