To address the issue of insufficient detection accuracy for small targets in dense scenes during pedestrian detection tasks, an improved pedestrian detection algorithm named CSTPF-YOLOv8 based on the YOLOv8 model was proposed. By integrating a convolution and attention fusion module into the backbone network, the algorithm enhanced the ability to extract features from key regions and improved the model's perception of target details. The feature fusion network was optimized to promote the effective integration of cross-scale features, thereby improving the model's adaptability and detection accuracy for targets of different sizes. Detection heads were added to the lower-level feature maps to capture small targets using the advantage of high resolution. Methods such as the Focal-CIoU boundary regression loss function were proposed to study small target pedestrian detection in dense scenes. The results show that compared to the basic YOLOv8 network, the CSTPF-YOLOv8 algorithm increases the mean average precision by 1.4% and 3.2% on the WiderPerson and CityPersons datasets, respectively, demonstrating the effectiveness of this experimental method.