To address the challenges in continuous sign language recognition, such as the difficulty of effectively integrating RGB features and skeleton features, and the low recognition accuracy of gestures that are visually similar but semantically different, a continuous sign language recognition model combining cross-modal attention and semantic awareness was proposed. Firstly, a dual-stream S3D network was constructed to extract spatiotemporal features from videos and keypoint sequences, respectively, and a cross-modal attention mechanism was used to fuse multimodal information, enhancing feature complementarity and temporal alignment. Next, a language-aware label smoothing strategy was proposed, generating soft labels based on the cosine similarity between tokens calculated from FastText pre-trained word vectors, incorporating semantic priors to distinguish gestures that were visually similar but semantically different. Finally, the entire model was jointly optimized with connectionist temporal classification(CTC) loss, smoothed classification loss, and visual branch auxiliary loss. Multiple comparison experiments were conducted on the Phoenix-2014, Phoenix-2014T, CSL-Daily, and ArabSign datasets. Experimental results show that the word error rates of the proposed method on these four datasets are reduced to 18.2%, 18.8%, 24.3%, and 47.6%, respectively.