Aiming at the need for a fast inference speed and excellent performance model for road defect detection on resource-constrained mobile devices, an improved YOLOv5-based road defect detection algorithm, YOLO-RGCS, was proposed. Before training, K-means + + clustering algorithm was used to generate pre-anchor boxes. Then, a lightweight vision transformer(RepViT) was introduced to eliminate and skip irrelevant computational and memory costs, improving inference speed. Subsequently, depthwise separable convolution blocks were added to reduce the model's computational load. Furthermore, an attention mechanism was incorporated to enhance the model's focus on multiple targets in a single image and to weaken background interference factors. Finally, network channel pruning was performed to remove some redundant network channels, improving the model's efficiency and speed. The test results of the model show that compared to the original model, the YOLO-RGCS model has a reduced volume of 14%, an increased detection speed of 11.8%, with Precision and recall improving by 5% and 9.8%, respectively, and mAP increasing by 2.7%. These results demonstrate the acceleration improvement of YOLO-RGCS, making it suitable for deployment on mobile devices or in low-cost environments for real-time road defect detection.