In the detection of internal defects in objects using phased array laser ultrasound (LUPA), noise interference can lead to inaccurate signal recognition, thereby reducing the accuracy of quantitative defect detection. To address this issue, a defect recognition method is proposed that integrates the Sparrow Search Algorithm-Weighted Permutation Entropy (SSA-WPE) optimized Variational Mode Decomposition (VMD) with the Vision Transformer model. A finite element model of LUPA is established to simulate the detection process. Ultrasonic echo signals of circular defects with different radii and depths, as well as rectangular cracks with different lengths and depths, are collected, and noise is added to simulate the actual detection environment. The normalized noisy signals are decomposed and reconstructed using the SSA-WPE-VMD method, and the reconstructed signals are then inverse-normalized to form the final reconstructed signals. Continuous Wavelet Transform (CWT) is applied to the reconstructed signals to generate time-frequency maps, which are subsequently fed into the ViT model for training, ultimately achieving quantitative defect detection. Experimental results show that, for both circular defects and rectangular cracks, the proposed method achieves an average accuracy of over 98% on both the test and validation sets, with an average AUC close to 1, validating the feasibility and effectiveness of the method.