A two-stage quantitative damage detection method based on difference method and improved particle swarm optimization algorithm was proposed to solve the structural damage problem of laminates during processing and service. Firstly, the structural damage element was defined by the method of reducing the elastic modulus of the element, and the modal parameters of the numerical model of the laminate under different damage conditions were obtained by analyzing the structural vibration characteristics. Secondly, an improved particle swarm optimization algorithm was used to quantitatively calculate the exact location and degree of damage. The proposed algorithm constructed the objective function through the flexibility matrix of the modal parameters of the tested structural model and the iteratively updated finite element model. The variation factor was introduced to make the algorithm effectively escape the local convergence. The MS operator greatly reduced the influence of the healthy element on the damage degree of the real damaged element. The results show that the improved particle swarm optimization algorithm can accurately identify the location and damage degree of the damaged elements in the composite laminates.