To solve the problem of insufficient extraction of sport features by dual stream networks in current action recognition, which leads to low recognition accuracy, a action recognition method based on sport feature enhancement two-stream networks was proposed to improve accuracy. The network was divided into spatial stream and temporal stream, with the same structure but different inputs. The input of the spatial stream network was a video frame sequence, while the input of the temporal stream network was a video frame difference sequence. The network structure used Resnet50 as the backbone network, replacing the 3 × 3 convolution with the proposed global sport feature module and local sport feature module, fully extracting video sport information, and finally combining spatial and temporal stream to output the results. The results show that the accuracy of the model on the UCF101 and HMDB51 datasets reaches 96.8% and 75.3%, which is superior to traditional algorithms.