To address the challenges in detecting surface defects in subway tunnels, such as blurred boundaries, multi-scale variations, slender shapes, and class imbalance, an improved SegFormer-ESF model was proposed. First, a parallel edge information enhancement branch was designed. Fixed Sobel operators and depthwise separable convolutions were used to extract and enhance gradient features. These edge features were adaptively fused with backbone features through a feature fusion module, which improved perception and localization of defect boundaries. Second, a directional strip convolutional attention module was embedded in the encoder. It employed 1×K and K×1 convolutions to model geometric structure and directional context of linear defects, such as cracks and leakage, thereby enhancing representation of slender objects. Then, a lightweight multi-scale pyramid feature fusion module was introduced to reconstruct the decoder, improving multi-scale feature integration while reducing model complexity. Finally, to mitigate class imbalance caused by low pixel proportion of defects, a combined Focal-Dice loss function was proposed, which jointly optimized hard example mining and overall region recognition. Experimental results show that SegFormer-ESF achieves 83.53%, 92.46%, and 97.61% on mIoU, mPA, and Acc, respectively, improving by 3.72%, 3.51%, and 0.46% over SegFormer-B0. In addition, the model has 3.620 M parameters and only 3.02 G computational cost, reduced by 2.56% and 55.46% compared with the baseline, respectively, achieving lightweight while improving segmentation accuracy. The FPS reaches 88.53 f/s, which can meet the demand for real-time detection, providing a feasible solution for intelligent identification and mobile deployment of tunnel defects.