The internal damage of the steel rail is serious, but the non-destructive testing B-display detection image has a lot of noise and noise, and the spatiotemporal distribution characteristics of different damages are not obvious, making it difficult to effectively identify. In response to this situation, a rail screw hole crack B-image recognition algorithm based on improved YOLOv8 was studied to improve the accuracy of intelligent identification of rail damage. Firstly, to reduce the missed detection of small damage targets, RepHGNetv2 network was used to optimize the YOLOv8 backbone network and improve the detection recall rate. Then, in order to improve the adaptability of the model to different types of damage detection, the detection head of YOLOv8 was replaced with Effientnet to improve the detection accuracy of the model. Finally, the LSKA attention mechanism was introduced into the SPPF module to enhance the model's anti-interference ability against noise signals and improve its accuracy. The actual line detection results have verified that the detection accuracy of the above model reaches 95.1%, the recall rate reaches 93.8%, and the average accuracy reaches 97.6%, which is improved compared to other commonly used algorithms.