Crack detection is crucial to maintaining the structural safety of buildings. In recent years, convolutional neural networks based on deep learning have provided new solutions for crack detection. However, this comes at the cost of huge computing resources, so there are problems of poor real-time performance and low detection efficiency in practical applications. To address this problem, a lightweight MSFC (multi-scale dynamic fusion convolution module) based on the U-Net architecture was proposed to improve the efficiency of crack segmentation. To verify the effectiveness of the proposed method, a dataset Crack2045 containing 2 045 crack images was constructed and experiments were conducted on this dataset. The experimental results show that compared with the original U-Net model, the model using the MSFC module reduces 78.51% of the parameters and 63.75% of the computational complexity while maintaining the same accuracy. At the same time, the MSFC module has a certain degree of generalization and can be seamlessly integrated into different semantic segmentation models. This study not only provides an efficient deep learning method for crack detection, but also provides new possibilities for model deployment in resource-constrained environments.