Pavement diseases caused by the synergistic effect of environment and load have an increasingly prominent impact on in-service performance and durability performance. It is difficult for the existing intelligent image recognition algorithms to achieve a balance between processing speed and computational complexity. Aiming at the requirements of fast, accurate, and real-time identification of road diseases, the road surface with serious damage in Shijiazhuang was taken on the spot, combined with the existing pictures, the data augmentation technology was used to construct the municipal road disease dataset, and a lightweight road disease identification network model, GEM-MobileNetV3, was proposed based on the MobileNetV3 network. Firstly, the Ghost module was used to replace the 1×1 convolution in the basic unit of the MobileNetV3 network. Then, combined with the improved efficient channel attention module (ECA), essential features of disease targets were extracted. Finally, the ReLU activation function in the shallow layer of the network was replaced by the Mish activation function with a stronger generalization ability to improve the overall performance of the model. The effectiveness of the new model was verified by ablation experiments and comparative experiments. Experimental results show that the accuracy of the new model reaches 96.33%, and the number of parameters and computations are reduced by 37.9% and 36%, respectively, compared with the MobileNetV3 model. The proposed new model can effectively reduce computational complexity while maintaining high recognition accuracy, which provides a new way to achieve high-accuracy real-time recognition on low-cost computing platforms.