Emotion recognition, as a key research topic in the fields of human-computer interaction, mental health assessment and social computing, has received extensive attention in recent years. However, it is difficult for traditional single-modal methods to capture complex emotional features comprehensively, and there are still many difficulties in effectively integrating multi-modal physiological signals such as temperature signal (TEM), electroencephalogram (EEG) signal and electromyography (EMG) signal. To solve these problems, a multi-modal emotion recognition model SE-CNN-LSTM was proposed, which integrated channel attention mechanism. The model first used the convolutional neural network (CNN) to extract spatial features from the input multimodal data, and then adaptively adjusted the weight of feature channels by introducing the squeeze and excitation (SE) attention mechanism. Enhance features were more important for emotion recognition. Finally, by taking advantage of the time series modeling advantage of the long short-term memory (LSTM) network, the dynamic changes of emotions in the time dimension were captured. The experimental results on the open data set show that the accuracy of the model reaches 98.85% and 98.61% on the titer and arousal binary classification task, and 98.12% on the quad classification task, respectively. The result provides a new idea for multimodal emotion computing and has potential application value in many fields.