To address boundary-detail loss and category confusion in RGB images and digital surface model (DSM) fusion for multimodal remote sensing image semantic segmentation, an encoder–decoder framework named HDFNet was proposed, in which hierarchical dual-frequency modeling was introduced. Wavelet decomposition was performed to decouple multimodal features into high-frequency structural components and low-frequency se-mantic components. DSM high-frequency information, which contains more stable structural cues, served as guidance for correcting RGB high-frequency features, and the aligned high-frequency structures were further used to enhance low-frequency semantic representations. The enhanced features were then reconstructed into the spatial domain through inverse wavelet transform for semantic segmentation prediction. The results show that HDFNet achieves mIoU scores of 84.29% and 86.67% and mF1 scores of 91.34% and 92.71% on the ISPRS Vaihingen and Potsdam datasets, respectively, outperforming conventional multimodal fusion methods. These results indicate that HDFNet can improve segmentation performance in boundary recovery and con-fusing-category discrimination.