Most of the current electronic signature handwriting recognition models mainly focus on time series features and ignore the correlation between dimensions, and the feature modeling ability of time series features is insufficient. To solve this problem, an electronic signature handwriting recognition model based on double-branch Transformer was proposed to realize high-precision recognition of electronic signature handwriting. Firstly, a dual-branch feature extraction structure of Transformer was designed to fully capture temporal features and channel features.Secondly, in order to realize multi-scale temporal feature learning, a multi-scale feature extractor was introduced in the temporal feature branch to enhance the encoder’s fine-grained learning of temporal features, and a feature recalibration module was introduced in the encoder to strengthen the learning of key points in the signature time sequence.In addition, a channel attention module encoder with a dual-view pooling mechanism was proposed in the channel feature branch to enhance the correlation between features of different dimensions.Finally, a dual-branch feature fusion module was designed to effectively fuse complementary temporal features and channel features using a hierarchical design of local enhancement first and global fusion later. The performance evaluation was carried out on the public datasets MSDS and MCTY. The EER(equal error rate) of the simulation scene 1 vs. 1 achieves 2.54% and 2.61% respectively, and the EER of the simulation scene 4 vs. 1 achieves 1.64% and 1.87% respectively. The experimental results show that the proposed method has higher recognition accuracy and has certain advantages in complex scenes.