In order to improve the high-temperature performance of asphalt, a variety of composite modified asphalts with different proportions of rubber and diatomite were prepared. The performance of various modified asphalt formulations was assessed through tests measuring penetration, softening point, ductility, and rotational viscosity. Subsequently, the optimal modifier blending ratios were determined using response surface methodology. The modification effects were evaluated by comparing them to rubber asphalt and rubber-SBS composite modified asphalt. The aging properties, high-temperature performance, microstructure, and reaction mechanisms of the rubber-diatomaceous earth composite modified asphalt were investigated through rotational thin-film oven aging tests, thermal gravimetric analysis, differential scanning calorimetry, Fourier-transform infrared spectroscopy analysis, and scanning electron microscopy analysis. The results indicate that the composite modified asphalt with 22% rubber and 4% diatomite exhibits the best overall performance. Compared to rubber asphalt and rubber-SBS composite modified asphalt, the rubber-diatomite composite modified asphalt demonstrates superior high-temperature stability. There is mainly physical blending between rubber powder, diatomite, and matrix asphalt. Asphalt and the two form a uniform and stable whole that is fixed to each other.