In high-speed transmission gear systems, the relative sliding velocity of gear teeth is significant, leading to an increase in the instantaneous temperature of the tooth surface. This temperature differential induces thermal stress, alters thermal deformation, and modifies the material properties of the gear teeth, subsequently affecting the time-varying mesh stiffness (TVMS) of the gear. To thoroughly investigate the intrinsic relationship among thermal deformation, friction, and time-varying meshing stiffness of gears, an improved method for calculating TVMS of gears was introduced, which incorporates friction and thermoelastic coupled deformation. This approach specifically addresses the non-involute errors in the gear tooth profile resulting from thermal deformation. The findings indicate that changes in the direction of friction force can induce a step effect in the TVMS within the single tooth meshing region. Furthermore, the TVMS of the gear increases with a higher friction coefficient. Thermal deformation due to temperature variations modifies the original involute characteristics of the tooth profile, thereby enhancing gear contact ratio. Although the elastic modulus of the material decreases with rising temperature, when considering the effects of friction and thermal deformation during gear meshing, the TVMS of the gear increases with temperature. Thus, the impacts of temperature characteristics and friction on gear transmission performance are significant and cannot be overlooked.