In response to the low detection accuracy and high model complexity of existing road damage detection algorithms in complex environments, a lightweight road damage detection algorithm named LDC-YOLOv5 (lightweight deformable convolution YOLOv5) was proposed based on YOLOv5. To address the complexity of real road surface damages, a lightweight feature extraction module was designed using Deformable Conv (deformable convolution) and Depthwise Conv (depthwise convolution) to replace the C3 module in the original network backbone, enabling convolutional kernels to focus on irregular crack damages and enhancing feature extraction for damage detection. To reduce algorithm complexity in the feature fusion stage, a lightweight feature fusion module was constructed using GhostConv to replace the C3 module in the original network neck, lowering network parameters and complexity. Additionally, to prevent missed detections caused by uneven lighting and shadow obstruction, a lightweight attention mechanism, TripletAttention, was introduced in the backbone network to improve the algorithm's understanding of damage information and context. Experiments conducted on the IEEE open dataset RDD2022 and the Kaggle open dataset Road Damage demonstrate that, compared to YOLOv5s, the proposed LDC-YOLOv5 achieves a 1.4% and 4.2% improvement in mAP50 on the two datasets, respectively, with only 67.6% of the model parameters of YOLOv5s.