High computational complexity and large parameter counts are often observed in existing video action segmentation models when long-term dependencies are handled. An action segmentation network balancing high precision and lightweight design was proposed to address these issues. A dual-branch parallel architecture fusing dilated attention mechanisms and channel convolutions was proposed based on experimental analysis. Weights were cooperatively learned by this dual-branch structure to achieve complementary advantages. A convolutional dilated attention mechanism was introduced. Local features were captured, and translation invariance was maintained. The high computational cost of traditional attention mechanisms was effectively addressed. The CDFormer (convolutional dilation transformer) model was constructed based on the Transformer architecture and the convolutional dilated attention mechanism. The results indicate that the accuracies of CDFormer on the 50Salads, GTEA, and Breakfast datasets are 88.13%, 80.14%, and 74.55%, respectively. Superior performance is demonstrated. The efficiency and superiority of the convolutional dilated attention mechanism in action segmentation tasks are thus verified.