A damage diagnosis method based on adaptive noise suppression wavelet transform (ANR-WT) and a multi-branch CTM model was proposed. First, the time-frequency map and multi-scale wavelet sequence of the signal were adaptively extracted using ANR-WT, and physical feature vectors were supplemented to form a multi-modal feature characterization. Furthermore, a cooperative three-branch network was designed: MobileNetV2 convolutional neural network branches extracted time-frequency image spatial features, Transformer branches captured the global dependence of wavelet series, and multilayer perceptron branches learned the internal laws of physical features. Ultimately, these three complementary features were fused. Cross-dataset generalization experiments demonstrated that the average diagnostic accuracy of this model reached 99.91% on the Case Western Reserve University (CWRU) dataset, while the highest diagnostic accuracy reached 98.20% on the Jiangnan University (JNU) bearing dataset. These results indicated that this method exhibits excellent performance and strong cross-domain generalization ability compared to traditional methods in rolling bearing damage diagnosis.