Cloud fluctuations cause variations in the concentrated energy flux density of tower solar power systems, and these variations produce thermal stress and fatigue in the receiver. To reduce the influence of clouds, accurate cloud identification is required. A cloud instance segmentation model based on YOLOv8 was proposed. The C2f module in the Neck was replaced with a multi-scale dilated attention (MSDA) module to achieve model lightweighting. Standard convolution layers were substituted with arbitrary kernel convolution (AKConv) to improve the extraction of cloud features. The spatial pyramid pooling fast (SPPF) module was upgraded to the SPPFCSPC structure to enable richer multi-scale spatial information extraction from sky images.The accuracy of the improved MAS-YOLO model’s target detection indicators is 75.8%. The recall of the target detection indicators is 65.5%, and the mean accuracy of the target detection indicators is 75.1%. The accuracy of the segmentation indicators is 79.9%, the recall of the segmentation indicators is 62.9%, and the mean accuracy of the segmentation indicators is 74.7%. The computational complexity is 54.6 GFLOPs. On the SWIMSEG dataset, MAS-YOLO was compared with other models. The results show that its recognition performance is better than that of the other models, and heat-map comparisons indicate that it extracts cloud features more effectively. This model provides methodological support for optimizing the performance of tower solar energy systems.