In response to the low detection accuracy and high model complexity issues of existing road damage detection algorithms in complex environments, a lightweight road damage detection algorithm, named Lightweight Deformable Convolution YOLOv5 (LDC-YOLOv5), is proposed based on YOLOv5. Firstly, to address the complexity of real road surface damages, a lightweight feature extraction module is designed using deformable convolution (Deformable Conv) and depthwise convolution (Depthwise Conv) to replace the C3 module in the original network backbone. This enables the convolutional kernels to focus on irregular crack damages, thereby enhancing the feature extraction capability for damages. Secondly, to tackle the high algorithm complexity in the feature fusion stage, a lightweight feature fusion module is constructed using GhostConv as a replacement for the C3 module in the original network neck part, reducing the network parameters and complexity. Additionally, to prevent missed detections caused by uneven lighting conditions and shadow obstruction, a lightweight attention mechanism called TripletAttention is introduced in the backbone network to enhance the algorithm's understanding of damage information and context. Finally, experiments conducted on the IEEE open dataset RDD2022 and the Kaggle open dataset Road Damage demonstrate that compared to YOLOv5s, the proposed LDC-YOLOv5 achieves a 1.4% and 4.2% improvement in mAP50 on the two datasets, respectively, with only 67.6% of the model parameters of YOLOv5s.