To address the issues of low precision and high model complexity in existing pavement defect detection algorithms, a lightweight automatic detection algorithm for pavement defects based on YOLOv11 was proposed, incorporating a Triplet Attention mechanism (TAD-YOLO11). Firstly, to overcome the limitation of the traditional spatial pyramid pooling-fast (SPPF) module, which solely focused on edge information while neglecting background context, a novel pooling scheme was proposed. This design leveraged adaptive average pooling and adaptive max pooling layers to integrate both global background information and edge details, thereby facilitating effective fusion of multi-scale feature information. Secondly, to enhance cross-dimensional interaction of feature data and compensate for the shortcomings of the original C2PSA module in the channel dimension, the triplet attention mechanism was incorporated into the original module to construct a new lightweight feature fusion module, C2PSA-TA. In the backbone network, the lightweight and efficient upsampling operator DySample was adopted to replace the traditional upsampling algorithm, with triplet attention added to better balance the training data and improve the algorithm’s ability to understand feature information. Finally, testing and validation were conducted on the public datasets RDD2022 and UAV-PDD2023. The results show that compared to YOLOv11n, TAD-YOLO11 exhibits superior performance, specifically, precision improves by 2.5% and 6.9%, respectively, mean average precision increases by 2% and 5.2%, respectively, while the model’s parameter count and computational load are reduced. Additionally, the processing speed is 1.25 times and 2.38 times faster than the baseline model.