In order to solve the problem of low accuracy of dense pedestrian detection in complex scenarios, the dense pedestrian detection algorithm CHD-DETR (crowd-human-detect-DETR) was developed by improving the RT-DETR algorithm. Firstly, a partially separated PSNET(pyramid scene parsing network) module was designed to improve the backbone network. This improves the network’s ability to extract small target features, and reduce the computational cost of the model. Secondly, a novel hypergraph aggregation module named HAM (hybrid aggregation module) was designed to capture the high-order and low-order feature relationships. By integrating this module with the concept of pyramid spatial pooling, an average hybrid network named AHNET(average hybrid network) was designed as the multi-scale feature fusion, effectively reducing the loss of detailed information. Finally, a convolutional high-efficiency layer aggregation CELA-Block (convolution ELAN block) module was designed to replace the RepC3 module, enhancing the extraction of local features and improving detection accuracy. The CHD-DETR model was evaluated through ablation experiments on the INRIA+Kaist and CrowdHuman datasets, and comparative experiments were conducted with mainstream algorithms. Compared with the RT-DETR baseline model, CHD-DETR achieved improvements of 1.7%, 3.3%, and 5.5% in mAP50, mAP50:95, and precision, respectively, on the INRIA+Kaist dataset. On the CrowdHuman dataset, it achieved increases of 1.0%, 1.1%, and 1.4% in mAP50, mAP50:95, and recall, respectively, while reducing computational cost by 10.3 GFLOPs. Experimental results show that the proposed CHD-DETR model exhibits good performance in dense pedestrian detection under complex scenarios.