To address the problems of dim light, image affected by lamp light and difficulty in detecting small targets at long distance in highway tunnel, an improved YOLOv5 algorithm for vehicle and personnel detection in highway tunnel was proposed. Firstly, Gaussian mixture clustering was used to obtain a set of anchor boxes that match the dataset targets better, improving the detection accuracy of the model. Secondly, CARAFE operator was introduced in the feature fusion to enlarge the receptive field and reduce the loss of feature detail information in the upsampling process. Finally, by inserting coordinate attention (CA) into the network, the model's ability to extract features from different positions of the image is further enhanced. To verify the effectiveness of the algorithm, experiments were conducted on the Zhejiang Wenli Expressway Tunnel dataset, and the results show that: the mean average precision (mAP) of the proposed algorithm reaches 95.7%, which is 3.8% higher than the original model, and it can achieve more accurate detection for small targets at long distances and targets affected by severe lighting, providing an effective solution for vehicle and personnel detection in highway tunnels under complex environments.