Vibratory compaction stands as a crucial engineering technique in highway subgrade compaction works, rapidly and effectively enhancing the strength and overall stability of the subgrade materials. Additionally, the continuous compaction control technology based on vibratory compaction offers novel technical means for monitoring and managing the quality of subgrade compaction. To investigate the vibratory roller response characteristics during the compaction process and the suitability of commonly used harmonic ratio-based compaction quality real-time evaluation indicators at different compaction stages, the interaction between the vibratory roller and the subgrade system was analyzed in this study. The dynamic models for the contact and detachment conditions were established respectively, and the step function δ(F_c) was used to unify the dynamic equations for both scenarios. Simulink was utilized to simulate these dynamic equations, and the simulation results were analyzed to assess the response characteristics of vibratory compaction and the applicability of indicators. It is indicated that the subgrade compaction process can be divided into four stages by comparing the variation cycle T_c of the contact force between the vibratory roller and the soil with the excitation period T₀ : T_c = 0, T_c = T₀ , T_c = 2T₀ and T_c > 2T₀. During the T_c = 0 stage, integer multiples of harmonic signals are observed, and the amplitude of these harmonics increases with the stiffness of the soil. At the T_c = 2T₀ stage, signals of 0.5 times the harmonic are observed.. Comparative analysis reveals that the CCV (compaction control value) and THD (total harmonic distortion) indicators exhibit better applicability throughout the entire compaction process compared to the CMV (compaction meter value) indicator. Furthermore, the RMV (resonance meter value) indicator accurately reflects the compaction process reaching stage three (T_c = 2T₀).