In order to improve the fuel economy of plug-in hybrid electric vehicle(PHEV) and solve the problems of feature extraction distortion, unstable training processes, and low local optimization accuracy in the standard twin delayed deep deterministic policy gradient(TD3) algorithm, an energy management strategy based on an improved TD3 algorithm was proposed in this paper, in which attention mechanisms and residual network structures were integrated into the Actor network, and an adaptive noise decay model was introduced, and a guided reward function was constructed. The simulation results show that the fuel consumption of the proposed strategy is reduced by 7.29% and 23.90% compared with the standard TD3 algorithm and the charge depleting-charge sustaining strategy, respectively. In addition, better fuel economy is achieved and the stable battery state of charge is maintained, and the engine operating range is optimized. Therefore, the energy-saving potential and control robustness of PHEV are significantly enhanced by the improved TD3 strategy.