Exploring the impact of various hole shapes on the acoustic emission properties of rocks and the fracture mechanisms within rock structures containing holes is of utmost importance, as it enables the detection of fracture progression in rock engineering and the prediction of instability and failure in defective rocks. The mechanical properties, acoustic emission evolution patterns, precursor characteristics of failure, and failure mechanisms of red sandstone samples with different hole shapes were investigated through uniaxial compression tests and acoustic emission systems. The results indicate that the compressive strength, elastic modulus, and strain energy of the intact specimen are approximately 1.4 times, 1.3 times, and 1.7 times greater, respectively, than those of the specimen containing holes. There is a mutation point in the evolution of the multifractal spectrum of AE at 86%~95% of the peak stress, where the width undergoes a transition from an average low value to a sudden increase before and after the mutation point. This change is accompanied by an increase in fluctuation range, from small to large. The AE waveform characteristics are distinguished by the presence of dual main frequency bands. As the sample approaches failure, a significant number of low-frequency and high-amplitude signals, along with high-amplitude and high-amplitude signals, are generated. These peculiar trends in acoustic emission can be used as precursors to the critical instability of red sandstone. Based on the correlation between the main frequency of the acoustic emission signal of the rupture event and the fracture mechanism of the rock, the failure mechanism of the intact and square cavity red sandstone is mainly characterized by tension-shear mixed failure, and the shear failure of circular cavity red sandstone is revealed, which effectively avoids the subjectivity of the RA-AF failure mode classification method, and the research results can provide certain guiding significance for engineering design and optimization.