In order to investigate the meshing of high speed helical gears, firstly, the dynamic equation of rotor-bearing system with multiple degrees of freedom was constructed by finite element method, and the natural frequency of gear meshing was calculated. Then, the displacement response was calculated by Newmark-β method. Secondly, KISSsoft was applied to establish the geometric model and modify the tooth shape in both profile and direction. The transient analysis of helical gear pairs before and after modification was carried out by using ANSYS Workbench, and the comparative analysis was carried out in terms of equivalent stress, fatigue life and safety factor. The results show that the modified helical gear has reduced equivalent stress, increased fatigue life and improved safety factor, which proves the feasibility of the modified method and provides a reference for the design of high-speed helical gear meshing research.