The collapse of hazardous rock masses is among the most destructive geological disasters globally, and the quantitative evaluation of its instability risk is the core of engineering risk management. To address this issue, This study proposes a simulation and risk assessment framework for the deterioration of hazardous rock masses based on structural dynamics. First, the Duhamel integral is utilized to dynamically simulate the vibration signals induced by internal fracture propagation. Subsequently, 8 dynamic monitoring indicators are extracted from four dimensions: time, frequency, energy, and modal domains, to establish a four-level risk assessment interval. At the data processing stage, an SVD-based vibration signal denoising method is proposed. A game theory combinatorial weighting approach is then applied to integrate the subjective analytic hierarchy process (AHP) and the objective entropy weight method, which effectively resolves the limitations of single weighting methods and constructs an index weight optimization model. Finally, a quantitative risk assessment model incorporating damage time-varying characteristics is developed by introducing the dynamic correlation function and interval matter-element concepts based on improved extension theory. Freeze-thaw experiments on hazardous rock mass collapse indicate that evaluations using non-denoised signals resulted in 5 serious misjudgments. After denoising, the instability risk level of the rock mass successfully rose to Level 4 at the moment of collapse, with the risk level characteristic value reaching 3.752. The results quantitatively capture the evolution of instability risk prior to collapse, thereby verifying the necessity of vibration signal denoising and demonstrating the accuracy and effectiveness of the proposed quantitative evaluation model.