Traffic sign assisted recognition technology is becoming more and more important in the automatic driving system. Due to the different bearing capacity of different hardware equipment, it has become one of the research directions of various units and enterprises to make the model lighter while the performance is unchanged or better. In order to make the model lighter and improve the model recognition effect and detection speed, a lightweight traffic sign detection method based on SWimAM (a simple, parameter-free attention module add weight part for convolutional neural networks) designed YOLOv5 (you only look once version 5) was proposed. This method added iterable learning weights based on SimAM mechanism and changes the calculation method of internal weights. SWimAM module was proposed and C3 layer of YOLOv5 backbone structure was replaced with this module. The head part was fused with SE (squeeze-and-excitation networks) attention mechanism and the loss function was replaced by SIoU (soft intersection over union) to enhance the detection accuracy of the model and reduce the instability of the gradient. A data enhancement method of filtering Mosaic images was proposed to solve the non-uniformity of partial labels. On the final TT100K traffic sign data set, the recognition mean average precision of the improved YOLOv5s network increase by 2.5%, calculation speed increased by 7.33%, computational complexity decreased by 3.07%, and the number of references decreased by 9.27%. The mean average precision of Chinese traffic sign detection data set CCTSDB and German traffic sign detection data set GTSDB reached 94.9% and 94.7%, respectively, which verified that the model has good generalization.